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Overview 


Purpose  of  Research 

Engineered  Resilient  Systems  (ERS)  is  one  of  the  seven  DoD  Science  and  Technology  (S&T) 
Priorities.  The  ERS  program  is  evolving  a  framework  and  an  integrated,  trusted  computational 
environment  supporting  all  phases  of  acquisition  and  operational  analyses.  Resilient  systems 
necessitate  data-driven,  richly  informed  decisions.  Achieving  resilience  in  the  past  has  relied  on 
expertise  of  experienced  senior  decision  makers  using  deep  knowledge  of  their  systems  within 
their  context.  In  support  of  the  ERS  tradespace  analysis  goals,  this  research  is  motivated  by  the 
pressing  concern  in  the  defense-related  industry  and  government  of  the  aging  workforce.  The 
resulting  wave  of  impending  retirements  of  experienced  personnel  may  lead  to  lost  knowledge 
and  inability  to  recreate  past  systems,  let  alone  develop  new  ones  of  growing  complexity.  The 
concern  over  lost  expertise,  including  both  in-domain  as  well  as  systems-level  thinking  is  a  real 
one,  and  one  that  has  received  growing  attention.  Using  a  structured  technique,  it  has  been 
shown  that  novices  can  approach  expert-like  strategies  in  the  use  of  visual  tradespace 
exploration  for  decision-making.  Additionally,  tradespace  exploration  has  been  shown  to  be 
useful  as  a  "boundary  object"  fostering  cross-domain  conversations,  and  facilitating  decision 
making  for  complex  systems.  Such  tradespace  results  are  promising  in  representing  a  means  for 
capturing  and  transferring  expert  knowledge  and  skills.  In  this  project,  the  research  team 
gathered  expert  knowledge  and  synthesized  emerging  ERS-related  research,  toward  a  goal 
enabling  novices  to  have  expert-like  decision  capability  through  encoded  knowledge  and  data- 
driven  tradespace  analysis  framework  and  integrated  tool  suite. 


Work  Accomplished 

Task  1.  Knowledge  and  Information  Gathering.  The  research  team  investigated  current  and 
recent  past  briefings  and  literature  related  to  engineered  resilient  systems,  and  specifically  to 
tradespace  exploration.  Multiple  discussions  were  held  with  government  leaders,  university 
researchers,  practitioners.  Given  the  importance  of  tradespace  exploration  (TSE),  the  team 
focused  its  efforts  in  this  regard  given  the  limitations  of  time  and  resources  in  this  research 
project  to  date. 

Task  2.  Artifact  and  Knowledge  Coding.  Using  the  gathered  information  and  insights,  and  the 
ongoing  research  of  the  team  members,  the  key  ERS-relevant  artifacts  were  identified.  The 
research  team  mapped  these  artifacts  to  the  MPTs  derived  from  the  knowledge  and  information 
gathering  performed  in  Task  1. 

Task  3.  Exploration  Case  Study.  This  task  was  initially  intended  to  conduct  an  exploratory  case 
study  to  identify  tradespace  exploration  artifacts  in  practice  and  how  they  relate  to  knowledge 
goals  within  ERS.  Unfortunately  delays  in  data  availability  cut  short  the  intended  timeframe 
across  which  to  perform  this  case  study.  As  an  alternative  approach,  the  research  team  was  able 
to  generate  some  observations  based  on  limited  data  availability  to  two  Navy  tradespace 
exploration  activities  conducted  in  concert  with  the  ERS  Program. 
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Task  4.  Full  Scale  Case  Study  Design.  Based  on  the  results  of  the  information  gathering  and 
observed  exploration  case  studies,  the  research  team  developed  some  initial  requirements  for 
designing  a  larger  full  scale  case  study. 

Task  5.  Synthesize  Preliminary  Prescription  for  ERS  Artifacts  for  Knowledge  Capture/Transfer. 

Using  the  results  of  tasks  1-4,  the  research  team  enumerated  some  potential  gaps  and  findings, 
along  with  suggested  enablers  for  consideration  in  evolving  the  ERS  framework  and  tool  suite 
environment  in  support  of  the  its  tradespace  analysis  goals. 


Findings 

A  number  of  next  steps  were  identified  over  the  course  of  this  research  which  would  enhance 

and  enable  the  ERS  vision,  specifically  as  related  to  tradespace  exploration  activities.  These 

include: 

•  Efforts  to  begin  compiling  appropriate  knowledge  relative  to  the  core  constructs  identified  in 
this  projects  including  past  needs,  contexts,  constraints,  design  space,  performance  space, 
value  space,  performance  model,  value  model,  lessons  learned,  language,  case  examples,  and 
tools. 

•  Research  into  effort  vs.  confidence  tradeoffs  so  that  projects  can  scale  effort  on  various 
activities  within  the  TSE  process  to  match  their  needs  subject  to  available  resources. 

•  Development  of  fidelity  tradeoff  guidance  and  associated  tools  so  that  studies  can  scale  TSE 
implementation  appropriately. 

•  Explicit  incorporation  of  resilience-related  ilities  evaluation  into  the  ERS  architecture, 
including  model  libraries  and  decision  analyzer  toolset. 

•  Inclusion  of  value  models  along  with  performance  models  in  the  model  data  store. 

•  Continued  piloting  of  parts  of  the  ERS  associated  TSE  processes,  as  well  as  full  end-to-end 
studies. 

•  Continued  community  building  by  the  ERS  program,  including  offices  such  as  the  various  A9 
(e.g.  AF/A9,  OAS,  AFSPC/A9,  etc.)  and  other  entities  with  responsibilities  overlapping  with 
proposed  ERS  vision  and  capabilities 

•  Further  research  into  supporting  enabling  methods,  processes,  and  tools  that  can  facilitate 
TSE  knowledge  capture  and  reuse,  as  well  as  resource-effective  studies  that  can  quantify  and 
identify  resilient,  high  value  system  solutions  in  diverse  application. 


Research  Results 

The  research  team  gathered  expert  knowledge  and  synthesized  emerging  ERS-related  research, 
toward  a  goal  enabling  novices  to  have  expert-like  decision  capability  through  encoded 
knowledge  and  data-driven  tradespace  analysis  framework  and  integrated  tool  suite. 
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Next  Steps 


•  The  research  team  will  be  using  knowledge  and  information  gained  in  this  study  to  inform  its 
work  on  RT-122,  Interactive  Model  Centric  Systems  Engineering  (IMCSE)  and  RT-113  llities 
Tradespace  and  Affordability  (ITAP),  and  share  with  other  relevant  SERC  projects. 

•  The  research  team  will  be  using  the  results  of  this  investigation  toward  the  development  of  a 
publishable  paper  to  transfer  findings  to  the  broader  systems  community. 

•  The  specific  findings  will  be  shared  with  the  ERS  sponsor  and  its  technical  team  and  partners 
(see  Figure  1)  in  appropriate  discussions,  workshops  and  research  exchanges  to  potentially 
influence  and  extend  the  ongoing  initiatives  in  support  of  the  ERS  vision. 
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Figure  1:  ERS  Technical  Team  and  Partners1 


1  Holland,  J.P.,  Engineered  Resilient  Systems  (ERS),  Briefing,  May  21st,  2014 
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Introduction 


Engineered  Resilient  Systems  (ERS)  is  one  of  the  seven  DoD  Science  and  Technology  (S&T) 
Priorities.  The  ERS  program  is  evolving  a  framework  and  an  integrated,  trusted  computational 
environment  supporting  all  phases  of  acquisition  and  operational  analyses.  Resilient  systems 
necessitate  data-driven,  richly  informed  decisions.  This  research  is  in  support  of  ERS,  and 
specifically  focuses  on  tradespace  exploration  and  its  contribution  to  achieving  the  ERS  vision. 


Motivation 

Achieving  resilience  in  the  past  has  relied  on  expertise  of  experienced  senior  decision  makers 
using  deep  knowledge  of  their  systems  within  their  context.  In  support  of  the  ERS  tradespace 
analysis  goals,  this  research  is  motivated  by  the  pressing  concern  in  the  defense-related  industry 
and  government  of  the  aging  workforce.  The  resulting  wave  of  impending  retirements  of 
experienced  personnel  may  lead  to  lost  knowledge  and  inability  to  recreate  past  systems,  let 
alone  develop  new  ones  of  growing  complexity.  The  concern  over  lost  expertise,  including  both 
in-domain  as  well  as  systems-level  thinking  is  a  real  one,  and  one  that  has  received  growing 
attention2.  Using  a  structured  technique,  it  has  been  shown  that  novices  can  approach  expert¬ 
like  strategies  in  the  use  of  visual  tradespace  exploration  for  decision-making.  Additionally, 
tradespace  exploration  has  been  shown  to  be  useful  as  a  "boundary  object"  fostering  cross¬ 
domain  conversations,  and  facilitating  decision  making  for  complex  systems.  Such  tradespace 
results  are  promising  in  representing  a  means  for  capturing  and  transferring  expert  knowledge 
and  skills. 

Over  time,  the  knowledge  of  a  practicing  engineer  progresses  across  a  spectrum,  from  factual  to 
conceptual  to  procedural  to  meta-cognitive,  with  these  levels  reflecting  a  growing  level  of  depth 
to  the  knowledge  content.  One  of  the  key  differences  between  novices  and  experts  is  the  ability 
for  "meta-cognitive"  thinking.  The  "meta-cognitive"  level  encompasses  a  higher  order 
knowledge  of  being  "self-aware"  of  the  knowledge  itself  (e.g.  "thinking  about  thinking").  Experts 
develop  this  skill  after  years  of  experience,  increasingly  focusing  on  the  deeper  levels  of 
knowledge,  and  requiring  them  to  "question  their  assumptions"  in  order  to  understand  the  limits 
of  their  perceptions,  cognitions,  and  conclusions. 

Not  all  engineers  can  achieve  this  type  of  skill3,  and  even  if  they  did,  it  may  take  a  long  time  to 
develop.  As  a  consequence  of  time,  complexity,  and  expertise  constraints,  often  engineers  must 
rely  upon  models,  methods,  processes,  and  tools  (MPTs)  to  encapsulate,  abstract,  analyze,  and 
synthesize  solutions  to  their  problems.  Particularly  attractive  to  resource-constrained 
practitioners  are  "tools,"  which  appear  to  have  an  attractive  ratio  of  effort  to  apply  versus  benefit 
obtained.  In  this  context,  a  "tool"  is  something  that  codifies  knowledge  as  an  "automated  or 


2  Lamb,  C.M.T.,  Collaborative  Systems  Thinking:  An  exploration  of  the  mechanisms  enabling  team  systems  thinking, 
Doctor  of  Philosophy  Dissertation,  Aeronautics  and  Astronautics,  MIT,  September  2009 

3  Rhodes,  D.H.,  Lamb,  C.T.  and  Nightingale,  D.J.,  "Empirical  Research  on  Systems  Thinking  and  Practice  in  the 
Engineering  Enterprise,"  2nd  Annual  IEEE  Systems  Conference,  Montreal,  Canada,  April  2008. 
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semi-automated"  set  of  steps  in  a  process,  thereby  "enhancing  process  performance  efficiency"4. 
The  automation  aspect  is  what  appears  to  lead  to  reduced  effort  to  apply,  as  opposed  to  the  time 
required  when  working  from  first  principles  to  solve  problems,  for  example. 

An  example  of  tool  use  is  in  the  area  of  tradespace  exploration.  Tradespace  exploration  is  the 
quantitative  comparison  of  large  numbers  of  system  alternatives  to  develop  knowledge  of  costs 
and  benefits,  as  well  as  their  relative  feasibility.  Using  a  structured  technique,  it  has  been  show 
that  novices  can  approach  expert-like  strategies  in  the  use  of  visual  tradespace  exploration  for 
decision-making5 6 7 8 9.  Additionally,  tradespace  exploration  has  been  shown  to  be  useful  as  a 
"boundary  object"  fostering  cross-domain  conversations,  and  facilitating  decision  making  for 
complex  systems6  7 .  Such  tradespace  results  are  promising  in  representing  a  means  for  capturing 
and  transferring  expert  knowledge  and  skills. 

In  fact,  "data-driven  tradespace  exploration"  has  been  explicitly  described  as  one  of  five  key  areas 
within  the  ERS  program8  9.  More  generally,  solutions  that  could  deliver  similar  benefits,  but  at 
lower  cost,  would  not  only  help  to  alleviate  the  financial  challenges  facing  government  today,  but 
also  increase  the  efficiency  of  US  industry  in  an  increasingly  competitive  global  marketplace. 

The  US  Department  of  Defense  has  recognized  the  potential  benefits  of  tradespace  exploration 
as  strongly  enabling  "Data  to  Decisions"  and  "Engineered  Resilient  Systems"  (ERS),  two  out  of 
seven  key  OSD  Science  and  Technology  Priorities  for  Fiscal  Years  2013-17  Planning10. 

The  benefits  of  an  ERS  approach  go  beyond  finding  potential  solutions  improving  cost-benefit 
efficiency.  In  particular,  an  engineer  must  expand  the  space  of  considered  alternative  solutions 
to  encompass  not  only  technical  aspects,  but  also  the  potential  contexts  in  which  they  might 
operate.  In  their  vision  of  what  it  means  to  be  an  Engineer  of  2020,  the  National  Academy  of 
Engineering  described  engineering  systems  professionals  with  the  knowledge  and  skills  who  can 
better  confront  significant  societal  problems,  rife  with  the  challenges  of  changing  contexts  (e.g.. 


4  Turner,  R.,  Shull,  F.,  Boehm,  B.,  Carrigy,  A.,  Clarke,  L.,  Componation,  P.,  Dagli,  C.,  Lane,  J.,  Layman,  L.,  Miller,  A., 
O'Brien,  S.,  Osterweil,  L.,  Sabados,  D.,  and  Wise,  S.,  "Evaluation  of  Systems  Engineering  Methods,  Processes  and 
Tools  on  Department  of  Defense  and  Intelligence  Community  Programs,"  Final  Technical  Report  SERC-2009-TR-004, 
Systems  Engineering  Research  Center:  New  Jersey,  December  2009,  p.  11. 

5  Wolf,  D.,  An  Assessment  of  Novice  and  Expert  Users'  Decision-Making  Strategies  during  Visual  Trade  Space 
Exploration,  MS  Thesis,  Mechanical  Engineering,  Penn  State  University,  May  2009 

6  Ross,  A.M.,  McManus,  H.L.,  Rhodes,  D.H.,  and  blastings,  D.E.,  "Revisiting  the  Tradespace  Exploration  Paradigm: 
Structuring  the  Exploration  Process,"  AIAA  Space  2010,  Anaheim,  CA,  September  2010 

7  Ross,  A.M.,  McManus,  H.L.,  Rhodes,  D.H.,  and  blastings,  D.E.,  "A  Role  for  Interactive  Tradespace  Exploration  in 
Multi-Stakeholder  Negotiations,"  AIAA  Space  2010,  Anaheim,  CA,  September  2010 

8  Neches,  R.,  "Engineered  Resilient  Systems:  A  DoD  Science  and  Technology  Priority  Area,  Overview  Presentation," 
June  2012,  p.  8 

9  Neches,  R.  and  Madni,  A.  M.,  "Towards  Affordably  Adaptable  and  Effective  Systems,"  Systems  Engineering  Journal, 
doi:10.1002/sys. 21234,  first  published  online  19  October  2012,  pp.  1-11 

10  SECDEF  Memorandum,  Science  and  Technology  (S&T)  Priorities  for  Fiscal  Years  2013-17  Planning,  April  19,  2011. 
Washington,  DC.  OSD  02073-11 
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technologies,  social  conditions,  and  policies)11.  In  order  to  meet  this  vision,  the  set  of  ERS  artifacts 
that  could  make  up  the  Engineer  2020  repertoire  must  incorporate  a  diverse  set  of  considerations 
that  allow  him  to  appropriately  match  methods  to  problem  types  and  use  cases  of  tomorrow. 

The  most  significant  benefits  to  accrue  from  the  success  of  the  proposed  research  is  in  mitigating 
or  avoiding  the  risks  identified  earlier:  "failure"  due  to  solving  the  wrong  problem,  and  "lost 
value"  due  to  fixating  on  inferior  solutions.  Additionally,  the  ERS  approach  has  the  potential  for 
identifying  more  efficient  allocation  of  resources  across  time  and  alternative  solutions.  ERS 
artifacts  that  effectively  capture  and  transfer  expert  knowledge  will  enable  less  experienced 
engineers  to  be  more  "meta-cognitive"  and  therefore  enable  junior  engineers  to  make  better 
decisions,  potentially  mitigating  the  impact  of  a  high  skill  workforce  shortage. 

One  objective  of  this  research  project  was  to  identify  how  ERS  artifacts  (and  other  supplemental 
knowledge  management  means)  can  act  as  an  expertise  transfer,  enabling  junior  engineers  to 
perform  at  the  level  of  more  experienced  engineers.  Tradespace  exploration  was  selected  as  the 
specific  focus  of  this  study. 


Insufficiencies  in  Current  Practice 

Insufficient  consideration  of  alternative  solutions  has  been  cited  as  a  risk  for  "lost  value"  in  the 
US  Air  Force.12  In  fact,  the  Government  Accountability  Office  (GAO)  has  found  that  "programs 
that  had  a  limited  assessment  of  alternatives  tended  to  have  poorer  outcomes  than  those  that 
had  more  robust  [analysis  of  alternatives]"  13.  In  some  cases,  service  sponsors  are  identifying  a 
preferred  solution  or  a  narrow  range  of  solutions  early  on.  Some  AoAs  are  conducted  on 
compressed  timeframe  to  meet  planned  milestones.  As  a  result,  this  "short  changes 
comprehensive  assessment  of  risks  and  preclude  effective  cost,  schedule,  and  performance 
trade-offs  from  taking  place  prior  to  beginning  development"  (quote?).  AoA  needs  to  occur  early 
enough  to  inform  decisions  before  program  initiation.  Findings  indicate  more  specific  AoA 
guidance  may  be  necessary  to  ensure  that  AoAs  meet  their  intended  objectives  and  provide  an 
in-depth  assessment  of  alternatives. 

The  implication  of  these  findings  is  that  tradespace  methods,  which  explicitly  consider  a  large 
number  of  alternatives,  are  well-suited  to  address  the  risk  of  "poor  outcomes"  (e.g.  "lost  value), 
thereby  increasing  the  likelihood  of  discovering  superior  solutions.  Tradespace  methods  have 
been  shown  to  be  useful  in  identifying  higher  value  efficient  solutions,  as  well  as  exploring  the 


11  National  Academy  of  Engineering,  "Engineer  of  2020:  Visions  of  Engineering  in  the  New  Century,"  Washington, 
DC:  The  National  Academies  Press,  2004,  pp.  1-118 

12  National  Research  Council,  "Pre-Milestone  A  and  Early-Phase  Systems  Engineering:  A  Retrospective  Review  and 
Benefits  for  Future  Air  Force  Acquisition,"  The  National  Academies  Press:  Washington,  DC,  2008,  pp.  1-150 

13  GAO,  "Defense  Acquisitions:  Many  Analyses  of  Alternatives  have  not  Provided  a  Robust  Assessment  of  Weapon 
System  Options,"  GAO-09-665,  September  2009,  pp.  1-41. 
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impact  of  various  contextual  constraints,  such  as  budget  caps  and  changes  in  available 
technology.14 


Scope:  Tradespace  Exploration 

Tradespace  exploration  is  a  technique  for  evaluating  a  large  number  of  alternatives  in  order  to 
generate  knowledge  and  insights  into  tradeoffs,  including  costs  and  benefits.  For  a  traditional 
system,  key  system  design  tradeoffs  are  made  in  the  initial  phase  of  the  lifecycle,  although 
tradespace  exploration  can  be  conducted  throughout  a  system  lifecycle.  Researchers  have 
demonstrated  tradespace  exploration  methods  as  effective  for  gaining  knowledge  of  the  design 
space15-16.  These  methods  enable  enumeration  of  alternative  concepts,  and  evaluation  and 
selection  of  promising  architectures  that  can  be  further  investigated.  Model-based  tradespace 
exploration  enables  comparison  of  a  much  larger  number  of  system  designs  than  is  possible  with 
heuristics  and  qualitative  approaches,  or  by  using  the  traditional  "point-based"  approach  in 
picking  a  design  followed  by  successive  refinement.  By  exploring  the  tradespace,  in  full  or  in  part, 
one  can  gain  knowledge  of  possible  system  performance,  relative  to  stakeholder  needs,  often 
described  as  "value",  or  benefit  at  cost.  Exploring  tradespaces  can  reveal  key  tradeoffs  as  well 
as  multi-stakeholder  preference  agreements  and  conflicts. 
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Figure  2  Notional  tradespace  exploration  flow 


Tradespace  exploration  leverages  "models"  in  order  to  evaluate  multiple  alternative  solutions 
(spanned  by  the  design  space)  in  terms  of  decision  criteria  (e.g.  "attributes",  KPPs,  MOE,  etc.)  and 
other  factors.  In  support  of  top  level  acquisition  decisions,  these  can  be  codified  as  costs  and 


14  Ross,  A.M.  and  Hastings,  D.E.,  "The  Tradespace  Exploration  Paradigm,"  INCOSE  International  Symposium  2005, 
Rochester,  NY,  July  2005,  pp.  1-13 

15  Ibid. 

16  Ross,  A.M.,  McManus,  H.L.,  Rhodes,  D.H.,  and  Hastings,  D.E.,  "Revisiting  the  Tradespace  Exploration  Paradigm: 
Structuring  the  Exploration  Process,"  AIAA  Space  2010,  Anaheim,  CA,  September  2010. 
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benefits  for  orienting  the  decision  maker  to  the  evaluated  alternative  set.  Tradespace 
exploration  encourages  "exploration"  as  opposed  to  optimization,  explicitly  empowering  the 
human-in-the-loop  analyst  or  decision  maker  to  investigate  the  implications  of  the  relationship 
between  inputs  and  outputs  (via  the  model  and  constraints/constants)  as  well  as  the  definition 
of  the  problem  (i.e.  the  proposed  input/design  space  and  the  proposed  output/value  space).  In 
spite  of  the  apparent  simplified  representation  of  points  on  a  two-dimensional  scatterplot  as 
seen  above,  the  actual  tradespace  data  set  can  be  much  larger  and  representing  high  dimensional 
tradeoffs  of  a  large  set  (e.g.  thousands,  or  millions,  or  more)  of  alternatives.  Oftentimes 
tradespace  exploration  generates  data  through  computer-based  models,  allowing  for  the 
automated  assessment  of  the  alternatives  set  and  helping  the  analyst  avoid  the  imposed 
limitations  of  focusing  on  a  few  point  designs  at  a  time. 

The  key  processes  of  tradespace  exploration  are  illustrated  in  Figure  3  below17.  These  include: 

•  Generation:  The  definition  of  the  design  space  (spanned  by  a  design  variable  set,  for  example) 

•  Enumeration:  The  definition  of  particular  alternatives  from  the  design  space 

•  Sampling:  The  definition  of  a  particular  subset  of  alternatives  from  the  enumerated  set 
intended  for  evaluation 

•  Evaluation:  The  evaluation  of  sampled  alternatives  via  model(s)  in  terms  of  desired  output 
metrics  (e.g.  benefits  and  costs  as  specified  by  stakeholders) 

•  Exploration:  The  intentional  investigation  of  relationships  and  patterns  between  the  input 
space  and  outputs  space,  resulting  in  knowledge  and  insights  for  the  analyst/decision  maker 

•  Selection:  The  decision  on  one  or  more  alternatives  as  "answering  the  question"  posed  by 
the  study,  for  example  providing  the  "best"  benefit  at  a  given  cost  across  considered  use 
contexts 


17  Ross,  A.  and  Rhodes,  D.H.,  "A  System  of  Systems  Tradespace  Exploration  Method,"  Chapter  5  in  Modeling  and 
Simulation  Support  for  System  of  Systems  Engineering,  ed.  Rainey,  L.  and  Tolk,  A.,  Wiley  and  Sons,  in  press  2014. 
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Figure  3  Key  processes  and  constructs  of  tradespace  exploration  (from  Ross  and  Rhodes  2014  in  press) 


These  key  processes  are  used  in  most  tradespace  exploration  activities,  although  they  may  be 
referred  to  by  different  terms. 
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Phase  1  Research 


In  phase  1  of  the  research,  the  team  conducted  knowledge  and  information  gathering  and 
collected  key  terms  that  form  the  basis  of  a  preliminary  lexicon  for  the  ERS  TSE  community. 


Knowledge  and  Information  Gathering  Findings 

The  research  team  gathered  knowledge  and  information  by  several  means.  The  current  and 
recent  past  of  ERS  program  was  examined  through  discussions  with  leadership  and  review  of 
recent  briefing  material.  Tradespace  exploration  specific  findings  were  gathered  through 
informal  interviews,  literature  review,  and  specific  discussions  with  subject  matter  experts. 
Findings  draw  from  a  number  of  organizations,  programs  and  projects  involved  in  tradespace 
exploration  policy/practice  setting,  research,  and  application  of  methods,  processes  and  tools 
(MPTs).  The  constraints  of  this  project  did  not  allow  for  an  exhaustive  investigation,  but  did 
produce  a  number  of  key  insights,  observations  and  finding.  The  findings  suggest  directions  for 
further  study  and  investigation. 


Stakeholder  Perspectives  on  Challenges,  Gaps  and  Issues 

In  order  to  enable  the  support  of  key  decisions,  especially  ones  that  are  resilient  to  dynamic 
uncertainties,  it  is  important  to  frame  the  problem  from  the  perspective  of  the  decision  makers. 
This  research  has  taken  a  multi-prong  approach  to  characterizing  the  challenges,  gaps,  and  issues 
from  a  decision  point  of  view.  This  includes  leveraging  recent  research  that  has  gathered 
stakeholder  expectations  and  needs  for  the  process  of  using  tradespace  exploration  in  decision 
making,  to  enable  evidential  reasoning.  The  mental  model  for  decision-making  using  evidential 
reasoning  is  described  below. 


Artifacts/Processes  Decision  Phase 

Problem  Statement  - -  Decision  Makers 


Potential  Solutions  -  '  -  Enumeration 

! 


Evidential  Reasoning 

Evaluation 

p  Data/Evidence  -i  * - 

-  Data  Generation 

“Models” 

Visualizations  * — 

-  Data  Exploration 

I - Tradeoffs  — - -*  - - 

-  Decision-making 

T 


“Best”  Solutions  - *  Selection 

J 

The  Solution  * -  Decision  Made 


Figure  4.  Mental  model  for  decision  making  involving  "evidential  reasoning" 

Figure  4  above  attempts  to  capture  the  essence  of  the  mental  model  for  motivating  the  questions 
below.  The  left  side  of  the  figure  lists  the  "tangible"  artifacts  and  processes  involved  with  making 
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a  decision,  while  the  right  side  of  the  figure  lists  the  principal  phases  for  decision  making: 
Enumeration,  Evaluation,  and  Selection.  The  left  side  begins  with  a  problem  or  needs  statement 
that  encapsulates  what  is  trying  to  be  solved  (for  example,  the  need  for  some  system  capability). 
Potential  solutions  are  enumerated  (by  engineers,  for  example)  through  some  formal  or  informal 
process.  Evidential  reasoning  involves  the  use  of  models  and/or  data  that  form  the  basis  of 
evidence  that  shows  whether  any  or  all  of  the  potential  solutions  can  solve  the  problem  or  meet 
the  needs.  Visualizations  may  be  used  to  represent  the  data  or  evidence  to  help  in  uncovering 
the  degree  to  which  potential  solutions  meet  the  needs.  Trade-offs  may  be  necessary  where 
solutions  do  not  meet  all  needs  and  dominate  all  alternatives.  The  potential  solutions  that 
satisfactorily  pass  the  trade-off  and  reasoning  process  become  the  "best"  solutions  from  which 
a  final  solution  is  selected.  Addressing  these  "decision  phases"  is  important  for  ensuring 
ultimately  good  decisions  will  be  made  in  the  engineering  of  resilient  systems.  Current  and 
emerging  advances  in  tradespace  exploration  MPTs  can  enable  the  process  across  all  phases 
including  constructs  for  accelerating  evidential  reasoning. 


Stakeholder  Interviews:  Expectations  and  Needs  for  TSE  for  Decision  Support 

A  recent  study  involved  the  elicitation  of  expectations  and  needs.  Stakeholder  interviews  were 

conducted  within  a  large  government  agency  to  gather  perspectives  on  the  current  practices  and 

needs  where  TSE  would  potentially  be  applied.  These  are  summarized  below. 

Decision  makers 

Q:  Do  you  have  a  specific  person,  group,  or  type  of  decision  maker  in  mind  to  benefit? 

A:  There  is  a  need  for  having  the  ability  to  bridge  the  gap  between  the  engineer  and  the 
decision  makers.  There  are  two  types  of  "decision  makers".  There  are  internal  and 
external  decision  makers.  The  internal  decision  makers  are  usually  engineers  and  the 
external  decision  makers  are  usually  non-engineers.  The  internal  decision  makers  have  a 
hard  time  communicating  with  the  external  decision  makers  and  in  vise-versa. 

Q:  Are  there  different  types  of  decision  makers? 

A:  Yes,  each  of  the  decision  makers  (external)  comes  in  with  different  needs  and  each  can 
have  different  objectives  which  can  range  from  cost,  schedule,  sustaining  an  industrial 
base,  sustaining  current  capability,  proposing  a  specific  system  which  fits  their  needs,  or 
any  type  of  performance  metric.  The  internal  decision  makers  attempt  to  come  up  with  a 
system  which  satisfies  the  external  decision  makers  and  convey  the  finding  to  the  external 
decision  makers. 

Q:  How  many  decision  makers  typically  decide  on  system  selection? 

A:  The  community  can  involve  anywhere  from  4  very  senior  decision  makers  to  100  lower 
level  offices. 

Q:  Do  these  decision  makers  work  in  parallel  or  sequence? 

A:  The  external  decision  makers,  for  the  most  part  work  in  parallel.  They  work  on 
supporting  the  system  they  believe  to  be  the  best  for  their  needs.  The  lower  level  decisions 
are  made  in  sequence  and  the  engineers  work  with  the  decision  makers  to  come  to  a 
solution. 
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Enumeration 


Q:  How  are  alternative  solutions  enumerated?  (e.g.,  reuse  of  past  systems,  formal  process,  etc.) 
A:  The  initial  stage  of  the  design  starts  with  prior  domain  knowledge  and  kicking  off  a 
study  to  consider  multiple  designs.  The  multiple  designs  are  gradually  weeded  out 
through  analysis,  comparing  the  performance  metrics  the  external  decision  makers  would 
like  to  have.  The  multiple  designs  are  carried  through  various  studies  until  there  are  only 
a  few  designs  left  for  the  external  decision  makers  to  choose  between.  There  tend  to  be 
problems  with  decision  makers  having  a  specific  system  they  would  like  to  have.  No 
amount  of  analysis  is  able  to  steer  them  in  a  different  direction  from  their  system. 

Q:  How  many  alternative  systems  are  typically  considered? 

A:  At  the  beginning  of  the  decision  process,  there  are  dozens  of  designs  considered.  After 
a  down  selection  process,  done  internally,  a  few  designs  are  presented  to  the  external 
decision  makers. 

Q:  How  many  alternative  scenarios  are  typically  considered? 

A:  Scenarios  can  be  defined  by  user  interests  and  can  correspond  to  five  or  more  scenarios. 

Evaluation 

Q:  What  is  the  level  of  fidelity  used  for  evaluating  alternatives? 

A:  The  models  used  are  medium  fidelity  models  and  tend  to  be  more  precise  than  accurate. 
This  is  due  to  the  assumptions  going  into  the  model  so  results  are  relative  and  not 
absolute. 

Q:  How  does  this  vary  based  on  technical  phase  of  decision  making  (i.e.,  "Pre-phase  A"  vs.  "CDR" 

vs.  "PDR"  level  decisions)?  How  is  the  level  of  fidelity  for  modeling/simulation  determined? 

A:  As  the  design  process  progress,  the  fidelity  of  the  model  increases.  This  increase  is  for 
two  main  reasons:  more  is  known  about  the  system,  assumptions  are  tightened  up  and 
more  specifics  of  the  system  are  known  and  start  to  get  modeled. 

Q:  Is  the  focus  on  "conceptual  design"-level  decision  making  appropriate? 

A:  It  is  difficult  to  keep  the  decision  maker  at  the  right  level  of  abstraction.  We  would  like 
to  keep  the  external  decision  makers  focused  on  the  performance  metrics  they  value  and 
the  tradeoffs  between  the  different  performance  metrics  so  the  internal  decision  makers 
can  help  design  a  system  specific  to  the  external  decision  maker's  needs.  It  is  hard  to  keep 
the  external  decision  makers  from  trying  to  offer  engineering  advice  and  attempting  to 
design  the  system  for  the  engineers. 

Q:  What  are  the  current  representations  in  use  for  visualizing  data? 

A:  From  engineer  to  engineer,  the  visualization  of  the  data  is  through  Excel  data  sheet, 
charts,  and  graphs  in  Power  Point.  The  technical  level  of  the  data  is  not  limited.  As  the 
data  goes  up  the  chain  of  the  decision  makers,  all  data  is  presented  through  Power  Point 
and  the  amount  of  technical  data  presented  is  condensed  and  only  the  most  pertinent 
data  is  presented.  As  the  data  goes  up  through  the  decision  makers,  the  technical  data  is 
condensed  and  also  attempted  to  put  in  a  format  which  is  hopefully  conveyed  to  the 
decision  maker  with  less  technical  background.  If  the  decision  makers  have  any  questions, 
and  cannot  be  answered  then  and  there,  the  question  travels  down  to  the  engineers  to 
answer  and  then  it  goes  back  up  the  chain  of  decision  makers. 
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Q:  What  are  the  current  tools  used  to  represent  data? 

A:  The  tool  used  to  convey  data  are  charts ,  graphs,  and  tables  put  in  a  Power  Point 
presentation. 

Q:  What  are  the  current  tools  used  to  explore  data? 

A:  Charts,  graphs,  and  tables  are  used  to  explore  the  data.  If  a  guestion  comes  up  not 
presented  in  the  charts,  graphs,  or  tables  then  the  guestion  is  answered  at  a  later  time. 

Q:  What  are  the  current  processes  for  exploring  data? 

A:  Charts,  graphs,  and  tables  with  an  iterative  process  on  the  decision  maker  getting 
together  to  get  the  guestion  answered  from  the  previous  meeting.  Sometimes  the 
decision  makers  will  ask  the  same  guestion  to  different  people  to  get  a  different  opinion 
on  the  guestion. 

Q:  What  are  the  current  processes  for  evaluating  alternatives? 

A:  The  engineers  attempt  to  consider  different  metrics  and  come  up  with  figure  of  merit. 
The  figure  of  merit  is  usually  a  performance  versus  cost  chart.  The  main  issue  with 
evaluating  alternatives  is  the  different  external  decision  makers  comes  in  with  an  idea  of 
which  system  they  would  like.  Once  the  engineers  propose  their  finding  and  their 
recommendation  on  the  system/s  there  is  usually  a  discrepancy  with  the  external  decision 
makers'  system.  This  in  turn  causes  many  more  iterations  of  analysis  because  the  external 
decision  maker  believes  their  system  was  not  evaluated  properly. 

Selection 

Q:  How  do  decision  makers  ask  questions? 

A:  Questions  are  asked  orally  and  by  the  time  they  are  written  down  and  given  to  the 
engineering,  the  guestion  is  distorted  (the  game  phone  call).  The  engineer  will  answer  the 
guestion  tersely. 

Q:  To  what  degree  do  non-technical  factors  affect  decisions? 

A:  Many  different  factors  affect  the  debate  on  what  is  motivating  the  external  decision 
makers.  At  the  end  of  the  day  it  is  usually  performance  metrics  which  can  be  measured 
discretely  which  wins  the  debate  over  a  fuzzy'  eguation. 

Q:  What  are  the  current  processes  for  communicating  decisions? 

A:  PowerPoint,  word  of  mouth,  and  email. 

Q:  Are  there  any  major  constraints  about  which  we  should  be  aware? 

A:  Schedule  is  a  fact  of  life  and  there  seems  to  be  politics  which  gets  involved. 

Summary.  Based  on  the  results  of  the  questions,  there  appears  to  be  a  need  to  provide  a  decision 
making  framework  that  can  be  used  across  multiple  decision  makers,  as  well  as  continuously  "up 
the  chain"  of  detail  from  low  level  engineering  up  to  senior,  high  level  decision  makers. 
Consistency  and  responsiveness  to  questions  can  accelerate  "buy-in"  and  communication  in 
order  to  facilitate  decision  making.  The  state  of  the  practice  involves  the  use  of  tables  and  Power 
Point,  which  is  inherently  static  and  not  responsive  if  external  decision  makers  introduce  new 
questions  during  reviews  and  meetings.  Quantitative  metrics  are  preferred,  as  well  as  the  ability 
to  consistently  aggregate  data  from  low  levels  of  analysis  up  to  high  level  performance  questions. 
Important,  non-technical  factors  will  tend  to  be  under-considered  if  not  represented  in  a 
concrete  fashion. 
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Stakeholder  Feedback  on  Benefits  of  Using  a  TSE  Approach 


A  government  user  of  the  tradespace  exploration  approach  discussed  some  of  the  problems  and 

limitation  with  traditional  trade-off  analysis.  This  individual  identified  the  following  benefits  of 

TSE  as  opposed  to  the  more  traditional  process  of  design: 

•  Forces  attention  on  value-driving  design  tradeoff  decisions,  rather  than  assumed  technical 
factors  (e.g.  prior  experience-derived  design  drivers,  or  reuse  of  design  drivers  from  prior 
studies) 

•  Reveals  design-value  tradeoffs  not  apparent  with  assessing  only  a  few  point  designs 

-  By  evaluating  a  large  number  of  points,  one  can  begin  to  see  the  patterns  of  relationships 
between  different  inputs  (i.e.  design  choices)  and  outputs  (i.e.  performance  and  cost) 

-  Interactive  tradespace  exploration  allows  decision  makers  to  test  their  mental  model  and 
develop  intuition  of  complex  system  tradeoffs 

•  Facilitates  cross-domain  communication  on  key  system  decision  factors 

-  The  tradespace  constructs  are  domain  neutral  (i.e.  cost-benefit  coordinates),  which  are 
mutually  understandable  and  focus  conversation  on  understanding  "why"  rather  than 
"what" 

•  Provides  an  ability  to  discover  compromise  solutions 

-  Fosters  identification  of  "good"  designs  per  decision  maker,  as  well  as  highlighting 
tensions  between  decision  makers 

-  Experts  often  unable  to  find  "suboptimal"  solution  that  may  be  better  compromise  across 
stakeholders  (i.e.  each  domain  expert  can  propose  an  80%  good  enough  solution  in  their 
own  domain,  but  impossible  to  guess  if  that  is  also  80%  in  other  domains) 

•  Is  a  structured  means  for  considering  large  array  of  possible  futures  for  discovering  robust 
systems  and  strategies 

-  Explicit  enumeration  of  design  space  and  context  space  leads  to  conversations  about 
possible  variety  of  solutions  within  a  possible  variety  of  situations  and  provides  structured 
analytic  framework  for  quantifying  the  goodness  over  time 


State  of  the  Practice 

Three  broad  areas  of  state  of  the  practice  for  TSE  (not  exhaustive)  were  investigated  by  the 
research  team,  including  (1)  current  TSE  approaches  and  toolsets,  (2)  TSE  related  areas,  and  (3) 
open  challenges  for  TSE. 

The  research  team  has  interacted  with  other  research  groups  involved  in  tradespace  exploration 
research,  and  followed  work  in  the  literature  and  in  technical  exchanges. 
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Academia 


Many  tradespace  exploration  toolsets  have  been  developed  over  the  years  within  academia.  For 
example,  the  ARLTrade  Space  Visualizer  (ATSV)18,  a  tradespace  data  visualization  tool  developed 
at  Applied  Research  Laboratory  at  Penn  State  University.  The  ATSV  toolset  continues  to  evolve, 
using  the  concept  of  "visual  steering"  to  drive  human-directed  optimization  within  a 
tradespace19.  Additionally,  the  Penn  State  team  has  integrated  the  concept  of  "value  robustness" 
into  some  of  their  work20.  Other  approaches  and  toolsets  from  various  organizations  were  also 
reviewed  to  look  for  insights  on  how  data  was  generated  and  displayed.  These  examples  include 
work  by  Georgia  Tech  using  GT-FAST  for  the  DARPA  F6  Program21,  as  well  as  the  Aerospace 
System  Design  Lab22,  and  JPL23. 

There  is  a  growing  effort  to  use  theory-based  approaches  to  develop  effective  tradespace 
exploration  constructs.  The  work  of  Stump  et  al.  uses  the  design  by  shopping  paradigm  whereas 
user  preferences  on  what  makes  a  good  design  emerges  through  interaction  with  the  possible 
alternatives  represented  in  a  multi-dimensional  tradespace.  Much  of  the  existing  work  appears 
to  extend  from  the  operations  research/optimization  and  decision  analysis  fields. 

In  the  knowledge  investigation  an  important  area  of  tradespace  exploration  is  in  accommodating 
different  types  and  levels  of  users.  The  work  of  Wolf  et  al.  on  novice  vs.  expert  users  is  an  example 
of  this  type  of  study24.  Overall,  there  is  a  growing  body  of  work  in  tradespace  exploration  that 
can  be  leveraged  as  the  development  of  the  ERS  TSE  architecture  and  toolsets  move  forward. 

In  particular,  in  recent  years,  there  has  been  growing  interest  in  using  a  "value-centric" 
perspective  for  framing  and  evaluating  tradespaces25,26.  This  is  in  contrast  to  a  "cost  versus 
performance"  perspective  and  results  in  the  ability  to  incorporate  the  impact  of  changing 
environments  and  uncertainty  on  the  decision  metrics.  Depending  on  the  case,  the  definition  of 


18  Stump,  G.,  Yukish  M.,  Martin  J.,  and  Simpson  T.  The  ARL  Trade  Space  Visualizer:  An  Engineering  Decision-Making 
Tool,  in  10th  AIAA/ISSMO  Multidisciplinary  Analysis  and  Optimization  Conference.  2004.  Albany,  NY,  Paper  No.  2004- 
4568. 

19  Stump,  G.M.,  Lego,  S.,  Yukish,  M.,  Simpson,  T.  W.,  Donndelinger,  J.  A.  (2009),  "Visual  Steering  Commands  for  Trade 
Space  Exploration:  User-Guided  Sampling  With  Example,"  Journal  of  Computing  and  Information  Science  in 
Engineering,  Vol.  9,  No.  4,  pp.  044501:1-10. 

20  Private  conversation  with  Professor  Timothy  Simpson,  July  2008  and  June  2009. 

21  Lafleur,  J  and  Saleh,  J.,  Exploring  the  F6  Fractionated  Spacecraft  Tradespace  with  GT-FAST,  AIAA  2009-6802,  2009. 

22  Mavris,  D.  N.,  &  Pinon,  O.  J.  (2012).  An  Overview  of  Design  Challenges  and  Methods  in  Aerospace  Engineering.  In 
Complex  Systems  Design  &  Management  (pp.  1-25).  Springer  Berlin  Heidelberg. 

23  Jones,  M.  and  Chase,  J.,  Conceptual  Design  Methods  and  the  Application  of  a  Tradespace  Modeling  Tool  for  Deep 
Space  Missions,  IEEE,  2008. 

24  Wolf,  D.,  Simpson,  T.,  and  Zhang,  X.  "A  Preliminary  Study  of  Novice  and  Expert  Users'  Decision-Making  Procedures 
During  Visual  Trade  Space  Exploration,"  Proceedings  of  the  ASME  2009  Int'l  Design  Eng.  and  Tech.  Conf.,  San  Diego, 
CA,  2009. 

25  Ross,  A.M.,  Multi-Attribute  Tradespace  Exploration  with  Concurrent  Design  as  a  Value-centric  Framework  for  Space 
System  Architecture  and  Design,  Dual  Master  of  Science  Thesis,  Aeronautics  and  Astronautics  and  Technology  and 
Policy  Program,  MIT,  June  2003. 

26  Brathwaite,  J.,  and  Saleh,  J.H.,  "Beyond  Cost  and  Performance,  a  Value-Centric  Framework  and  Pareto 
Optimization  for  Communication  Satellites,"  AIAA  Space  2009,  Pasadena,  CA,  September  2009. 
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"value"  varies  widely,  from  "net  present  value"  to  "multi-attribute  utility"27.  One  example  value¬ 
centric  TSE  approach  is  the  Multi-Attribute  Tradespace  Exploration  method,  developed  at  MIT 
and  expanded  over  14  years  of  research28- 29,30 

Figure  5  Example  academic  methods  and  tools  (non-exhaustive) 


Industry 


School 

Method(s) 

Tool{s) 

Georgia  Institute  of  Technology 

CATE33,  IRMA34 

RAVE35-36,  GTFAST37 

Georgia  Tech  Research  Institute 

FACT38 

Penn  State  /  ARL 

Design  by  shopping39 

ATSV40-41 

Massachusetts  Institute  of  Technology 

MATE42,  EEA4344 

IVTea/VisLab45,46 

... 

Tradespace  exploration  has  been  employed  in  practice  by  various  organizations  for  a  number  of 
years,  including  detailed  design,  across  domains  such  as  chip  design31.  The  recent  DARPA  META 
Program  wanted  to  leverage  advancements  in  industry,  especially  those  made  by  semiconductor 
manufacturers,  in  order  to  achieve  drastic  reductions  in  schedule  and  cost  of  development  for 
complex  DoD-class  systems32.  Specifically  META  aimed: 


33  Arruda,  J.,  Gavrilowvski,  A.,  Ahn,  B.,  Chae,  H.,  Spero,  E.,  and  Mavris,  D.N.,  "The  Capability  Assessment  and  Tradeoff 
Environment  (CATE)  for  Advanced  Aerospace  Vehicle  and  Technology  Assessment,"  Conference  on  Systems 
Engineering  Research  (CSER)  in  Procedia  Computer  Science,  Vol  28,  pp.  583-590,  March  2014. 

34  Engler,  W.O.  Ill,  Biltgen,  P.T.,  Mavris,  D.N.,  "Concept  selection  using  an  interactive  reconfigurable  matrix  of 
alternatives  (IRMA),  45th  AIAA  Aerospace  Sciences  Meeting  and  Exhibit,  2007. 

35  Daskilewicz,  M.  and  German,  B.,  "RAVE:  A  Graphically  Driven  Framework  for  Agile  Design-Decision  Support"  AIAA- 
2010-9033,  in  13th  AIAA/ISSMO  Multidisciplinary  Analysis  Optimization  Conference,  Fort  Worth,  TX,  Sep  2010. 

36  http://www.rave.gatech.edu/ 

37  Lafleur,  J  and  Saleh,  J.,  Exploring  the  F6  Fractionated  Spacecraft  Tradespace  with  GT-FAST,  AIAA  2009-6802,  2009 

38  http://gtri.gatech.edu/casestudv/gtri-software-helps-svstems-engineers-link-perform 

39  Stump,  G.M.,  Yukish,  M.,  Simpson,  T.  W.,  and  O'Flara,  J.  J.  Trade  Space  Exploration  of  Satellite  Datasets  Using  a 
Design  By  Shopping  Paradigm,  in  IEEE  Aerospace  Conference.  2004.  Big  Sky,  MT. 

40  Stump,  G.,  Yukish  M.,  Martin  J.,  and  Simpson  T.,  "The  ARL  Trade  Space  Visualizer:  An  Engineering  Decision-Making 
Tool,"  in  10th  AIAA/ISSMO  Multidisciplinary  Analysis  and  Optimization  Conference.  2004.  Albany,  NY,  Paper  No. 
2004-4568. 

41  http://www.atsv.psu.edu/ 

42  Ross,  A.M.,  Flastings,  D.E.,  Warmkessel,  J.M.,  and  Diller,  N.P.,  "Multi-Attribute  Tradespace  Exploration  as  a  Front- 
End  for  Effective  Space  System  Design,"  AIAA  Journal  of  Spacecraft  and  Rockets,  pp.  20-28,  Jan/Feb  2004 

43  Ross,  A.M.,  and  Rhodes,  D.H.,  "Using  Natural  Value-centric  Time  Scales  for  Conceptualizing  System  Timelines 
through  Epoch-Era  Analysis,"  INCOSE  International  Symposium  2008,  Utrecht,  the  Netherlands,  June  2008. 

44  Schaffner,  M.A.,  Wu,  M.S.,  Ross,  A.M.,  and  Rhodes,  D.FI.,  "Enabling  Design  for  Affordability:  An  Epoch-Era  Analysis 
Approach,"  Proceedings  of  the  10th  Annual  Acquisition  Research  Symposium-  Acquisition  Management,  April  2013. 

45  Ross,  A.M.,  McManus,  Fi.L.,  Rhodes,  D.FI.,  and  blastings,  D.E.,  "A  Role  for  Interactive  Tradespace  Exploration  in 
Multi-Stakeholder  Negotiations,"  AIAA  Space  2010,  Anaheim,  CA,  September  2010. 

46  Ross,  A.M.,  "Insights  from  a  Multisensory  Tradespace  Exploration  Laboratory  for  Complex  System  Selection," 
presented  at  MIT  SEAri  Research  Summit,  Cambridge,  MA,  Oct  2009. 
http://seari.mit.edu/documents/summit/2009/06-SEAriSummit09  RT  AMR. pdf 
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to  develop  model-based  design  methods  for  cyber-physical  systems  far  more 
complex  and  heterogeneous  than  those  to  which  such  methods  are  applied 
today;  to  combine  these  methods  with  a  rigorous  deployment  of  hierarchical 
abstractions  throughout  the  system  architecture;  to  optimize  system  design  with 
respect  to  an  observable,  quantitative  measure  of  complexity  for  entire  cyber¬ 
physical  systems;  and  to  apply  probabilistic  formal  methods  to  the  system 
verification  problem,  thereby  dramatically  reducing  the  need  for  expensive  real- 
world  testing  and  design  iteration.47 

One  of  the  outcomes  of  META  was  the  development  of  a  tool  suite  called  "CyPhy"  that  ended  up 
being  spun  off  into  a  for-profit  entity  called  CyDesign48.  CyDesign  aims  to  leverage  cloud-based 
infrastructure  to  implement  a  CyPhy-derived  software  that  enables  scalable  design  space 
exploration  by  users.  The  company  claims  to  enable  users  to  "explore  every  design,"  although  it 
is  unclear  how  the  design  space  is  defined49. 

The  United  Technologies  Research  Center  (UTRC)  process  for  rapid  design-space  exploration 
called  Architecture  Enumeration  and  Evaluation  (AEE)  involves  finding  all  feasible  architectures 
within  a  design  space,  based  on  a  set  of  design  rules.  Next,  at  each  abstraction  layer,  AEE  uses 
design  rules  to  rapidly  identify  the  sparse  set  of  feasible  architectures  from  within  the 
combinatorically  large  design  space  at  that  layer,  consisting  of  all  the  layer's  available  technology 
options  and  all  the  ways  in  which  they  can  be  interconnected.  Figure  6  show  the  visualization  of 
the  AEE  algorithm  searching  the  combinatorial  space.  According  to  Murray  et  al.50,  to  ensure  that 
ALL  feasible  architectures  are  found,  "AEE  considers  every  architecture  in  the  design  space,  but 
does  NOT  actually  build  or  visit  every  architecture.  AEE  builds  only  feasible  architectures,  and 
does  so  by  building  them  one  device  or  flow  at  a  time,  testing  only  the  rules  applicable  to  the 
added  or  deleted  device  or  flow  type.  Once  a  partial  architecture  is  found  to  be  infeasible,  an 
entire  subtree  within  the  overall  design-space  binary  tree  can  be  ruled  out  and  avoided.  The 
earlier  that  infeasibility  can  be  detected,  while  an  architecture  is  being  built,  the  larger  the 
subtree  that  can  be  avoided,  speeding  up  the  enumeration  process.  AEE  not  only  enumerates  the 


41  http://www.atsv.psu.edu/ 

42  Ross,  A.M.,  Hastings,  D.E.,  Warmkessel,  J.M.,  and  Diller,  N.P.,  "Multi-Attribute  Tradespace  Exploration  as  a  Front- 
End  for  Effective  Space  System  Design,"  AIAA  Journal  of  Spacecraft  and  Rockets,  pp.  20-28,  Jan/Feb  2004 

43  Ross,  A.M.,  and  Rhodes,  D.H.,  "Using  Natural  Value-centric  Time  Scales  for  Conceptualizing  System  Timelines 
through  Epoch-Era  Analysis,"  INCOSE  International  Symposium  2008,  Utrecht,  the  Netherlands,  June  2008. 

44  Schaffner,  M.A.,  Wu,  M.S.,  Ross,  A.M.,  and  Rhodes,  D.H.,  "Enabling  Design  for  Affordability:  An  Epoch-Era  Analysis 
Approach,"  Proceedings  of  the  10th  Annual  Acquisition  Research  Symposium-  Acquisition  Management,  April  2013. 

45  Ross,  A.M.,  McManus,  H.L.,  Rhodes,  D.H.,  and  Hastings,  D.E.,  "A  Role  for  Interactive  Tradespace  Exploration  in 
Multi-Stakeholder  Negotiations,"  AIAA  Space  2010,  Anaheim,  CA,  September  2010. 

46  Ross,  A.M.,  "Insights  from  a  Multisensory  Tradespace  Exploration  Laboratory  for  Complex  System  Selection," 
presented  at  MIT  SEAri  Research  Summit,  Cambridge,  MA,  Oct  2009. 
http://seari.mit.edu/documents/summit/2009/06-SEAriSummit09  RT  AMR. pdf 

47  http://cps-vo.org/group/avm/meta  [accessed  8/1/2014] 

48  http://cydesign.com/ 

49  http://cydesign.com/design-and-trade-space-exploration/ 

50  Murray,  B.,  et  al.,  META  II  Complex  Systems  Design  and  Analysis  (CODA),  United  Technologies  Corporation,  United 
Technologies  Research  Center,  AFRL-RZ-WP-TR-2011-2102,  August  2011. 
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feasible  architectures,  but  can  also  evaluate  partial  architectures  quantitatively,  and  use  their 
evaluation  to  reason  about  them,  for  example,  enforcing  quantitative  design  rules.  AEE  enables 
reasoning  about  network  architectures,  reaching  through  the  network  to  compute  necessary 
quantities  to  test  whether  requirements  are  met."  Alternate  AEE  implementations  have  also 
been  investigated. 
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Figure  6.  Visualization  of  UTRC's  AEE  Algorithm  Trace  through  Design  Space  Binary  Tree50 

A  set  of  TSE  relevant  tools  were  canvassed  by  Spero  et  al  (2014),  drawing  upon  surveys  conducted 
by  INFORMS51,52.  The  results  are  not  reproduced  here,  but  can  be  readily  found  online. 


Government/DoD 

Tradespace  exploration  is  conducted  in  different  manners  across  government  DoD.  Analysis  of 
Alternatives  (AoA)  are  related,  but  not  the  same  as  a  tradespace  exploration  study.  One  could 
conduct  a  TSE  study  to  support  an  AoA,  specifically  when  one  desires  addressing  the  gaps 
identified  by  GAO  outlined  earlier  (i.e.  insufficient  consideration  of  alternatives  and  premature 
focusing  on  preferred  alternatives).  The  Air  Force  has  provided  an  AoA  Handbook53  to  help  with 


51  Spero,  E.,  Avera,  M.P.,  Valdez,  P.E.,  and  Goerger,  S.R.,  "Tradespace  Exploration  for  the  Engineering  of  Resilient 
Systems,"  Conference  on  Systems  Engineering  Research  2014,  Redondo  Beach,  CA,  March  2014. 

52  http://www.orms-todav.org/surveys/das/das.html 

53  Office  of  Aerospace  Studies,  "Analysis  of  Alternatives  (AoA)  Handbook:  A  Practical  Guide  to  Analyses  of 
Alternatives,"  AFMC  OAS/A9,  www.oas.kirtland.af.rnil,  July  2008. 
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the  formulation  and  conduct  of  AoA,  including  AoA  process,  templates,  and  example  metrics  for 
alternatives  evaluation.  While  not  a  full  TSE  study,  this  structure  does  provide  a  good  starting 
point  for  conducting  such  as  study.  We  found  that  in  practice,  AoAs  often  do  not  necessarily 
follow  all  of  the  proposed  handbook  advice. 

In  spite  of  this,  there  are  a  number  of  efforts  in  DoD  that  approach  tradespace  exploration  in  the 
broad  sense.  For  example,  the  RDECOM  "whole  system  trade  approach"  "multiple  objective 
decision  analysis,"  links  together  individual  analyses  with  visualizations  that  enable  "drill  down 
capability"  and  "rapid  and  complete  understanding  of  the  trades  available  to  stakeholders."54 
Public  information  on  this  approach  is  limited,  so  the  researchers  were  unable  to  more  fully 
evaluate  its  strengths  and  limitations  vis  a  vis  the  current  state  of  practice  and  art. 


54  Chau,  D.,  and  Perriello,  D.,  "Utilizing  Model  Based  System  Engineering  to  look  at  the  Infantry  Squad  as  a  SoS 
architecture,"  RDECOM,  15  April  2014,  http://www.acq.osd.mil/se/webinars/2014  04  15-SoCECIE-Chau-Perriello- 
brief.pdf 
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Decision  support  model 
captures  and  synthesizes 
outputs  from  individual 
analyses  into  trade-space 
visualizations  designed  to 
facilitate  rapid  and  complete 
understanding  of  the  trades 
available  to  stakeholders  and 
provide  drill  down  capability 
to  supporting  rationale 
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Figure  7  Army  whole  system  trade  approach  (Chau  and  Perrieiio  2014) 


Cross-cutting 

As  described  in  Spero  et  al.100,  TSE  research  activities  across  government  and  academia  were 
investigated  to  look  for  commonality  in  process  execution.  Specific  TSE  projects  within  the 
government  were  also  investigated.  As  a  result,  a  set  of  twelve  common  process  steps  emerged 
in  their  work  (noting  that  these  were  not  tested),  as  shown  in  Figure  8.  These  authors  describe 
the  consolidation  of  processes  and  comparison  to  selected  projects  revealed  the  following:  "1) 
TSE  is  commonly  performed  in  steps,  as  assumed;  2)  there  exists  no  formal,  singular,  consistent 
process  for  performing  TSE  across  the  organizations  investigated;  3)  there  exists  no  singular  step 
within  TSE  processes  that  performs  the  action  of  "exploring"  a  tradespace  -  when  such  an  action 
is  performed  it  is  done  so  using  prerequisite  inputs  from  multiple  steps  while  simultaneously 
feeding  back,  and  forward,  exploration  results  to  the  decision  analysis  process;  and  4)  the  ERS 
DPs  both  omitted  steps  4  (subject  matter  expert  measures)  and  5  (stakeholder  value)". 
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Step  Description 

1  Determine  mission  scenario(s)  and  their  requirements,  and  keep  them  open  as  long  as  possible 

2  Identify  set  of  operational  performance  characteristics  and  high  level  system  design  variables  that  impact 
operational  requirements 

3  Apply  operational  engagement  models  against  various  mission  scenarios  and  threats  to  identify 
requirements,  MOP.  MOE.  and  other  performance  metrics 

4  Expert  knowledge  teams  determine  values  of  measures  for  given  mission  scenarios  and  requirements 

5  Break  down  stakeholder  values  into  roles,  attributes,  and  specific  tasks 

6  Generate  alternatives  that  meet  requirements  and  constraints,  and  map  stakeholder  values  to  system  design 
variables  using  scalable  multi-physics  based  modeling  design  tools 

Create  reduced-order  surrogate  models  to  show  iterative  ability  of  adjusting  scenarios  and  requirements  to 
physical  feasibility 

8  Qualitatively  or  quantitatively  rank  how  alternatives  meet  measures 

9  Perform  a  LCC  estimate  and  lifecycle  schedule  analysis  of  the  system 

10  Perforin  an  optimization  study  to  determine  the  optimum  feasible  space  that  meets  all  constraints  and  for 

each  course  of  action 

1 1  Determine  courses  of  action  based  on  optimal  feasible  space  and  perform  post-analysis  studies  (operational 
impact  and  gap  analyses) 

12  Perforin  case  studies  to  test  for  robustness  and  to  make  sure  that  the  alternative  solutions  are  resilient  in 
changing  operational  environments 

Figure  8.  TSE  Best  Common  Practice  Steps  (source:  Spero  et  al.100) 

The  findings  described  in  Spero  et  al.100  also  include  results  on  tool  functionality  and  tool 
attribute  categories. 


TSE  Related  Areas 

The  following  are  related  areas  to  TSE  and  are  not  meant  to  be  exhaustive  or  mutually  exclusive, 
but  merely  to  illustrate  the  existence  of  a  vast  relevant  literature  and  knowledge  base  that  should 
be  leveraged  in  TSE.  Each  of  these  areas  have  their  own  conferences  and  journals,  signifying  their 
identities  as  distinct  areas,  even  if  their  techniques  are  similar  or  complementary,  especially  in 
their  application. 


Systems  Engineering 

Trade  studies  are  a  key  activity  in  the  systems  engineering  process,  and  frameworks  for 
conducting  tradeoff  studies  continue  to  mature55.  A  trade  study  is  an  objective  evaluation  of 
alternative  requirements,  architectures,  design  approaches  or  solutions  using  identical  ground 
rules  and  criteria.  There  are  numerous  variations  of  the  trade  study  process,  but  have  the  basic 
steps  of  defining  the  objectives  of  the  study,  review  inputs  (including  constraints  and 
assumptions),  choosing  the  evaluation  criteria  and  their  relative  importance,  identifying 


55  Cilli,  M.V.,  and  Parnell,  G.S.,  "Systems  engineering  tradeoff  study  process  framework,"  24th  INCOSE  Int'l 
Symposium,  Las  Vegas,  NV,  June/July  2014. 
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alternatives,  assessing  performance  of  each  using  the  criteria,  comparing  results  and  then 
choosing  an  option.  The  limitation  of  a  trade  study  is  that  it  involves  the  trade-off  of  several 
options,  so  it  does  not  serve  the  same  purpose  as  tradespace  exploration.  Trade  studies  are  most 
effective  when  there  are  several  viable  options  to  consider  and  choose  from.  As  such,  once  a 
tradespace  exploration  study  identifies  several  good  designs  to  consider  in  a  more  detailed  study, 
a  tradeoff  study  process  can  be  applied. 


Statistics  and  Data  Science 

The  field  of  statistics  is  broad  and  there  is  no  need  to  summarize  it  here.  Suffice  to  say  statistics 
is  all  about  mathematical  treatment  of  data  analysis  and  is  clearly  relevant  to  any  data-driven 
endeavor,  including  tradespace  exploration.  A  big  caveat,  however,  is  that  the  data  often  used 
in  early  lifecycle  phases  for  a  system  may  be  created  via  modeling  and  simulation,  rather  than 
through  empirical  testing.  The  consequence  is  that  the  data  sets  may  be  "artificial"  (i.e. 
generated  via  human-made  abstractions  with  ground  truth  unknown)  and  therefore  statistical 
treatment  must  be  done  with  appropriate  caution  (i.e.  are  the  patterns  being  uncovered  "real" 
or  "artifacts",  both  challenges  in  analysis  of  natural  data,  but  especially  so  in  artificial  data.) 

The  "new"  field  of  data  science  merges  techniques  from  statistics,  computer  science,  math, 
machine  learning,  domain  expertise,  communication  and  presentation  skills,  and  data 
visualization.56  This  field  aims  to  leverage  statistics  and  math  techniques  with  a  more 
"engineering"  applied  approach,  grounded  in  making  practical  predictive  models  (e.g. 
recommendation  engines)  that  may  not  be  grounded  in  any  one  particular  theory,  but  is  "good 
enough"  to  generate  useful  results.  Data  science  claims  to  be  a  response  to  managing  the 
problem  of  "big  data"  (i.e.  how  to  make  sense  from  data  sets  that  are  large,  diverse,  dynamic, 
and  of  questionable  quality?)  Tradespace  exploration  activities  could  be  considered  to  be  a  type 
of  applied  data  science  where  the  problem  to  be  solved  involves  the  design  of  technical  systems. 


Applied  Decision  Support 

The  area  of  decision  analysis  (DA)  is  fundamentally  related  to  TSE  in  that  decision  analysis  aims 
to  apply  rigorous  quantitative  techniques  for  representing  stakeholder  desires  (i.e.  preferences) 
and  evaluating  and  ranking  a  set  of  alternatives.  A  key  difference  between  decision  analysis  and 
TSE  is  that  TSE  often  can  have  an  open  set  of  alternatives  (that  is,  the  set  of  evaluated  alternatives 
can  grow  or  shrink  over  time).  This  type  of  decision  problem  is  considered  "wicked"  within  the 
DA  community  since  it  is  a  very  difficult  problem  to  prove  correctness  for  different  ranking 
schemes.  In  a  sense  TSE  is  an  application  of  decision  analysis  and  one  of  the  outcomes  of  TSE  is 
to  support  decision  making  in  different  phases  of  the  system  lifecycle.  Research  and  practice 
within  the  DA  community  should  be  relevant  to  TSE,  especially  relating  to  the  evaluation  and 


56  O'Neil,  C.,  and  Schutt,  R.,  "Doing  Data  Science:  Straight  Talk  from  the  Frontline,"  Cambride:  O'Reilly,  2014,  p.  11. 
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selection  of  alternatives57'58'59.  It  is  worthwhile  to  note  that  it  has  been  argued  that  existing 
decision  methods  within  the  engineering  literature  do  not  sufficiently  adhere  to  the  rigorously 
derived  techniques  from  the  DA  community  and  that  an  opportunity  exists  to  rectify  this 
problem60.  In  particular,  Hazelrigg  argues  that  weighted  sum  of  attributes,  analytical  hierarchy 
process,  physical  programming,  Pugh  matrix,  quality  function  development,  Taguchi  loss 
function,  Suh's  axiomatic  design,  and  six  sigma,  all  suffer  from  undesirable  qualities  when  used 
as  selection  methods  in  engineering  design61.  Ten  "desirable  properties"  of  a  good  engineering 
design  selection  method  are  proposed,  which  motivated  the  development  of  Multi-Attribute 
Tradespace  Exploration,  a  TSE  approach  that  merges  techniques  from  DA  with  TSE62. 


Multi-Disciplinary  Optimization  and  Robust  Design 

A  variety  of  techniques  from  the  operations  research  and  optimization  literature  relate  to 
concepts  at  the  core  of  TSE.  These  include  concepts  such  as  decision  parameters  (i.e.  "design 
variables"),  objective  functions,  and  constraints.  Multi-disciplinary  optimization  (MDO)  explicitly 
incorporates  consideration  of  multiple  domain/discipline  specific  evaluation  techniques  (i.e. 
models)  to  determine  scores  for  alternatives,  and  potentially  uses  multiple  objectives  for 
evaluation  as  well.  This  version,  multidisciplinary,  multiple  objective,  optimization  (MDMOO) 
bears  a  strong  resemblance  to  techniques  inherent  in  TSE,  with  the  primary  difference  being 
emphasis  on  "optimization"  versus  "exploration."  While  this  may  appear  to  be  only  semantics, 
the  difference  is  essential.  Optimization  has  an  inherent  assumption  of  a  "correct"  answer,  which 
hinges  upon  the  definition  of  the  objective  set  as  well  as  a  non-empty  feasible  region  of 
alternatives  for  evaluation.  In  TSE,  the  objective  functions  themselves,  as  well  as  constraints,  and 
definition  of  the  alternatives  set,  are  all  open  sets,  that  is,  have  membership  that  can  change 
through  intervention  by  the  tradespace  explorer/analyst.  At  its  simplest,  TSE  can  turn  into  an 
optimization  exercise  when  the  analyst  is  no  longer  "exploring"  and  instead  locks  down  the 
objectives,  constraints,  evaluation  models,  and  alternatives  set.  A  great  deal  of  research  has  gone 
into  problem  formulation  and  solution  algorithms  for  optimization,  and  all  of  these  can  be 
leveraged  for  TSE.  See  for  example,  the  use  of  one  heuristic-based  optimization  technique 


57  Keeney,  R.L.,  and  Raiffa,  H.,  "Decisions  with  Multiple  Objectives:  Preferences  and  Value  Tradeoffs,"  Cambridge 
University  Press:  Cambridge,  1993. 

58  Parnell,  G.S.,  Driscoll,  P.J.,  and  Henderson,  D.L.  ed.,  "Decision  Making  in  Systems  Engineering  and  Management," 
John  Wiley  and  Sons:  Hoboken,  2008. 

59  Watson,  S.R.,  and  Buede,  D.M.,  "Decision  Synthesis:  The  Principles  and  Practice  of  Decision  Analysis,"  Cambridge 
University  Press:  Cambridge,  1987. 

60  Hazelrigg,  G.A.  (2003),  "Validation  of  Engineering  Design  Alternative  Selection  Methods,"  Engineering 
Optimization,  Vol.  35,  No.  2,  pp.  103-120. 

61  Ibid.  p.  116. 

62  Ross,  A.M.,  Multi-Attribute  Tradespace  Exploration  with  Concurrent  Design  as  a  Value-centric  Framework  for  Space 
System  Architecture  and  Design,  Dual  Master  of  Science  Thesis,  Aeronautics  and  Astronautics  and  Technology  and 
Policy  Program,  MIT,  June  2003. 
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(genetic  algorithms)  applied  to  the  conceptual  design  of  helicopters  (in  an  automated  fashion,  as 
optimization  has  a  tendency  to  pursue)63. 

In  the  mechanical  engineering  design  literature,  the  Method  of  Imprecision  (Mol)  was  developed 
to  treat  uncertainty  in  choosing  among  alternatives.  In  particular,  the  method  espouses  that 
fuzzy  set  theory  should  be  used  to  treat  imprecision  and  not  probabilistic  approaches,  as  there  is 
a  fundamental  difference  in  the  nature  of  what  is  unknown64.  The  semantic  differences  between 
this  distinction  and  the  meaning  of  uncertainty  in  the  context  of  design  were  not  resolved  in  the 
literature.  Mol  is,  however,  complementary  to  other  Robust  Design  methods  in  their  objective 
to  minimize  the  impact  of  unknown  factors  (e.g.  noise  or  random  fluctuation  in  the  traditional 
probabilistic  representation  of  uncertainty)  on  system  design  (e.g.  form  or  function).  Taguchi's 
Robust  Design  approach,  for  example,  seeks  to  optimize  a  design  by  driving  down  the  expected 
variation  in  a  design's  performance  variables  as  a  result  of  variance  in  inputs  (i.e.  a  "robust 
design. ..[is  one]  that  has  minimum  sensitivity  to  variations  in  uncontrollable  factors65.")  The 
concept  of  robustness  and  minimizing  sensitivity  to  changes  has  been  expanded  in  recent  years 
to  include  requirements  sensitivity  analysis66. 


Visual  Analytics 

According  to  the  National  Visualization  and  Analytics  Center  (NVAC)  chartered  by  the  US 
Department  of  Homeland  Security,  research  on  visual  analytics  is  critical  to  facilitating  advanced 
analytic  insights.  The  NVAC  defines  the  field  of  visual  analytics  as  "the  science  of  analytical 
reasoning  facilitated  by  interactive  visual  interfaces."  Focus  areas  in  this  field  include  analytic 
reasoning  techniques,  visual  representations  and  interaction  techniques,  data  representations 
and  transformations,  and  techniques  for  production,  presentation  and  dissemination  of  analysis 
results67. 

As  a  part  of  knowledge  gathering,  members  of  the  research  team  participated  in  the  IEEE  Visual 
Analytics  Conference  held  in  October  of  2009  to  review  research  and  talk  with  experts  from  the 
NVAC  and  research  groups  working  on  this  topic.  The  conclusion  is  that  there  is  a  growing  interest 
for  this  agenda,  but  as  yet,  there  is  little  evidence  of  empirically-derived  'best  practices'  for 
visualizing  complex  tradespaces.  The  work  to  date  is  primarily  a  study  on  how  various  data  has 
been  generated  and  represented,  most  often  in  either  text  processing,  geographic  information 
or  scientific  graphics.  The  body  of  literature  is  challenging  to  follow,  but  it  may  offer  some  specific 


63  Crossley,  W.A.,  and  Laananen,  D.H.  (1997),  "The  Genetic  Algorithm  as  an  Automated  Methodology  for  Helicopter 
Conceptual  Design,"  Journal  of  Engineering  Design,  Vol.  8,  No.  3,  pp.  231-250. 

64  Antonsson,  E.K.,  and  Otto,  K.N.,  "Imprecision  in  Engineering  Design,"  ASME  Journal  of  Mechanical  Design,  Vol. 
117(B),  pp.  25-32,  1995. 

65  Simpson,  T.W.,  "Handout  on  Robustness"  in  IE  466  -  Concurrent  Engineering  course  at  Penn  State  University, 
http://www.mne.psu.edu/simpson/courses/ie466/ie466.robust.handout.PDF 

66  Mavris,  D.  N.,  &  Pinon,  O.  J.  (2012).  An  Overview  of  Design  Challenges  and  Methods  in  Aerospace  Engineering.  In 
Complex  Systems  Design  &  Management  (pp.  1-25).  Springer  Berlin  Heidelberg. 
http://link.springer.com/chapter/10.1007/978-3-642-252Q3-7  1 

67  National  Visualization  and  Analytics  Center,  "Illuminating  the  Path:  the  R&D  Agenda  for  Visual  Analytics,"  2004. 
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strategies,  approaches  and  mechanisms  that  could  be  incorporated  into  tradespace  visualization 
work. 

To  date,  little  significant  work  in  this  community  is  oriented  toward  or  directly  applicable  to  the 
challenges  in  visualizing  complex  tradespaces.  Looking  at  others  types  of  visualizations,  however, 
may  offer  insights  in  the  work.  There  is  also  emerging  work  in  computational  strategies  that  will 
enable  interactive  data  exploration  that  merits  further  investigation  in  support  of  an 
environment  for  running  tradespace  exploration  sessions68.  The  role  of  value  models  in 
interactive  visualization  is  an  open  area  of  investigation69. 

In  the  synthesis  of  prior  tradespace  exploration  methods,  topics  such  as  differences  in  use 
between  novice  and  experienced  users24  may  have  comparable  papers  in  the  visual  analytics 
literature70.  In  this  latter  work  from  the  field  of  financial  analysis,  common  trends  have  been 
observed  in  tradespace  exploration  sessions  and  intend  to  study  through  experiments  in  the 
future  (for  example,  Dou  et  al.  observed  "novices  started  to  perform  "random"  explorations  after 
they  had  exhausted  the  strategies  that  they  learned  during  the  training  phase  ...  for  experts, 
however,  20  minutes  were  not  enough  for  some  to  perform  deeper  and  more  complex 
investigations").  While  preliminary  knowledge  is  being  generated  at  present,  the  current 
research  is  attempting  to  follow  some  of  these  common  threads  across  fields  and  knowledge 
areas. 


Open  Challenges  forTSE 

Several  open  challenges  were  identified,  which  illuminate  some  of  the  potential  boundaries 
between  the  "state  of  the  practice"  and  the  "state  of  the  art." 


Methods  for  Considering  Context  and  Time 

While  there  is  much  research  and  development  of  tools  in  fields  related  to  tradespace 
exploration,  much  of  the  work  involves  the  generation  and  exploration  of  design-performance 
data  given  a  set  of  requirements  and  mission  contexts.  A  key  challenge  identified  by  the  ERS 
program  is  the  fact  that  circumstances  change.  That  is,  factors  beyond  the  system  program  may 
change,  causing  the  perceived  success  of  the  system  to  change.  The  impact  of  these  changing 
factors  is  the  primary  motivation  to  pursue  resilient  development  processes  and  systems. 
Tradespace  exploration  must  be  expanded  in  scope  to  enable  resilience  in  both  the  development 
process  and  systems  and  to  support  assessment  of  such  resilience.  To  further  bolster  this 
concept,  the  literature  review  investigated  how  other  fields  incorporate  context  and  time.  This 


68  Endert,  A.,  North,  C.,  Chang,  R,  Zhou,  M.,  Position  paper:  Toward  usable  interactive  analytics:  Coupling  cognition 
and  computation,  KDD  Workshop  on  Interactive  Data  Exploration  and  Analytics  (IDEA),  2014. 

69  Ricci,  N.,  Schaffner,  M.A.,  Ross,  A.M.,  Rhodes,  D.H.,  and  Fitzgerald,  M.E.,  "Exploring  Stakeholder  Value  Models  Via 
Interactive  Visualization,"  12th  Conference  on  Systems  Engineering  Research,  Redondo  Beach,  CA,  March  2014. 

70  e.g.,  W  Dou,  D.H.  Jeong,  F.  Stukes,  W.  Ribarsky,  H.R.  Lipford,  R.  Chang,  "Comparing  Usage  Patterns  of  Domain 
Experts  and  Novices  in  Visual  Analytical  Tasks,"  ACM  SIGCHI  Sensemaking  Workshop,  2009. 
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examination  of  literature  from  other  domains  serves  to  validate  and  to  enrich  areas  of 
uncertainty  characterization  and  the  need  to  embed  these  considerations  into  TSE. 

While  dynamic  quantitative  analysis  methods  and  tools  exist  in  abundance  (e.g.  stochastic 
programming71,  time-expanded  decision  networks72,  etc.),  in  order  for  TSE  to  support  high  level 
decision  making  (e.g.  acquisition  of  systems)  it  must  be  both  transparent  and  versatile.  The 
transparency  need  is  so  that  decision  makers  (and  even  analysts)  without  deep  specialization  in 
a  particular  technique,  are  able  to  interpret  and  trust  results  of  analyses.  While  deep  technical 
analysis  may  be  essential  in  evaluating  alternatives  over  time,  the  communication  of  the  results 
must  be  transparent  to  the  assumptions  used  in  generating  the  data  so  that  users  understand 
how  to,  and  feel  comfortable  with,  incorporating  the  results  into  the  larger  set  of  considerations 
needed  for  making  decisions.  Additionally,  related  to  this  issue  is  the  need  for  versatility  in  the 
dynamic  method  so  that  it  can  incorporate,  or  at  least  remain  compatible  with,  a  variety  of 
metrics  (and  data  types)  so  that  decision  makers  are  free  to  incorporate  an  appropriate  mix  of 
evidence  for  making  their  decisions.  For  example,  economic  climate  might  impact  a  decision  and 
should  be  able  to  be  included  in  the  decision  process  even  if  the  dynamic  method  is  focused 
solely  on  evolution  of  technical  systems  and  performance.  It  is  for  these  reasons  that  TSE 
dynamic  methods  should  draw  upon  both  quantitative  (e.g.  operations  research/optimization- 
based)  methods,  as  well  as  more  qualitative  methods  that  may  be  more  narratively  satisfying  for 
human  decision  makers 

A  number  of  anticipatory  methods  were  investigated  in  this  research  effort.  Many  different  fields 
use  approaches  to  perform  anticipatory  practice.  One  example  is  environmental  scanning  in  the 
field  of  management,  involving  the  acquisition  and  use  of  information  about  events,  trends,  and 
relationships  in  an  organization's  external  environment,  the  knowledge  of  which  would  assist 
management  in  planning  the  organization's  future  course  of  action73.  Other  examples  include 
morphological  analysis74  which  has  been  applied  in  military  strategic  planning,  and  scenario 
planning75  used  in  business  development  and  sustainable  development  planning.  In  general,  it 
was  observed  that  legacy  methods  are  powerful  as  thinking  approaches  but  are  largely  graphical 
or  narrative,  though  some  use  quantitative  approaches  as  well  such  as  computational  scenario 
planning  (e.g.  in  the  policy  realm76).  It  was  found  that  some  researchers  (e.g.,  Ritchey)  are 
moving  toward  a  model-based  approach  to  scenario  development,  but  the  work  has  not 
progressed  significantly. 


71  Peter  Kail  and  Stein  W.  Wallace,  Stochastic  Programming,  John  Wiley  &  Sons,  Chichester,  1994. 

72  Silver,  M.R.,  and  de  Week,  O.L.,  "Time-Expanded  Decision  Networks:  A  Framework  for  Designing  Evolvable 
Complex  Systems,"  Systems  Engineering,  Vol.  10,  No.  2,  pp.  167-186,  2007. 

73  Choo,  C.,  "Environmental  scanning  as  information  seeking  and  organizational  learning",  Information  Research,  Vol. 
7  No.  1,  Oct.  2001. 

74  Ritchey,  T.,  "Scenario  development  and  risk  management  using  morphological  field  analysis:  research  in  progress," 
Proceedings  of  the  5th  European  Conference  on  Information  Systems,  Vol.  Ill,  1997,  pp.  1053-1059. 

75  Kazman,  J.,  Carriere,  S.  and  Woods,  S.,  "Toward  a  discipline  of  scenario-based  architectural  engineering,"  Annals 
of  Software  Engineering,  Vol.  9,  No.  1-4,  2000. 

76  Lempert,  R.J.,  S.W.  Popper,  and  S.C.  Bankes,  Shaping  the  Next  One  Hundred  Years-New  Methods  for  Quantitative 
Long-Term  Policy  Analysis,  2003,  Santa  Monica,  CA:  RAND.  pp.  187. 
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In  the  TSE  realm,  research  by  Ross  and  Rhodes  using  an  Epoch-Era  Analysis  (EEA)  approach  has 
demonstrated  a  means  for  encapsulating  context  and  time  into  modular  analysis  blocks  (called 
epochs  and  eras)77.  EEA  is  a  multi-step  approach  for  identifying  of  the  impact  of  changing 
contexts  and  needs  on  proposed  systems.  The  approach  combines  two  key  concepts:  "epochs" 
and  "eras."  The  "epochs"  part  refers  to  the  short  run  possible  futures  that  may  be  experienced 
by  a  system.  Described  as  a  pair  of  possible  contexts  and  needs,  the  epochs  encapsulate  one 
possible  environment,  among  many,  within  which  a  system  may  find  itself.  A  technically  sounds 
system  may  fail  when  confronted  by  unanticipated  or  harsh  epochs.  A  particular  time-ordered 
sequence  of  epochs  is  a  possible  system  era.  The  path  dependency  of  how  epochs  unfold  over 
time  may  have  a  large  impact  on  the  time-varying  success  of  a  system.  Strategies  for  delivering 
value  over  time  will  be  considered  for  a  system  across  possible  eras.  Along  with  Multi-Attribute 
Tradespace  Exploration  (MATE),  EEA  has  been  codified  as  the  Responsive  Systems  Comparison 
Method  and  has  been  demonstrated  on  several  case  studies  for  evaluating  the  impact  of 
changing  contexts  and  needs  over  time  on  tradespaces78,79. 

Additionally,  there  has  been  significant  interest  in  developing  solutions  that  are  able  to  remain 
"good"  in  spite  of  changes,  a  quality  sometimes  called  "robustness"  or  "flexibility"  in  a  system. 
Robust  design  has  been  traditionally  applied  to  the  manufacturing  of  systems,  but  also 
increasingly  upstream  to  the  concept  design  as  well80.  Flexibility,  likewise,  has  begun  to  be 
addressed  as  a  system  property  that  could  be  evaluated  during  tradespace  exploration81,82. 


Managing  the  Size  of  the  Design  and  Uncertainty  Spaces 

Another  key  challenge  facing  the  TSE  community  is  that  of  managing  the  sizes  of  the  evaluated 
spaces,  which  include  both  design  space  (potential  alternatives)  and  context  space  (potential 
environments  and  use  cases  of  the  alternative).  As  has  been  noted  in  the  literature,  despite 
increases  in  computing  performance  and  capabilities,  gains  in  addressing  the  problem  of 
evaluating  large  design  spaces  is  offset  by  increases  in  the  scope  and  scale  of  the  input  space,  as 
well  as  the  details  of  the  evaluation  models  themselves.  This  problem  is  reflective  of  the 
inherently  infinitely  large  possibility  space  confronted  during  open-ended  design.  The  "art"  of 
system  design  is  in  the  abstraction  of  the  actual  potential  system  alternatives  into  a 
representation  that  is  large  enough  to  span  a  reasonable  region  of  interest  in  the  design  space. 


77  Ross,  A.M.,  and  Rhodes,  D.H.,  "Using  Natural  Value-centric  Time  Scales  for  Conceptualizing  System  Timelines 
through  Epoch-Era  Analysis,"  INCOSE  International  Symposium  2008,  Utrecht,  the  Netherlands,  June  2008 

78  Ross,  A.M.,  McManus,  H.L.,  Rhodes,  D.H.,  Hastings,  D.E.,  and  Long,  A.M.,  "Responsive  Systems  Comparison 
Method:  Dynamic  Insights  into  Designing  a  Satellite  Radar  System,"  AIAA  Space  2009,  Pasadena,  CA,  September 
2009. 

79  Fitzgerald,  M.E.  and  Ross,  A.M.,  "Mitigating  Contextual  Uncertainties  with  Valuable  Changeability  Analysis  in  the 
Multi-Epoch  Domain,"  6th  Annual  IEEE  Systems  Conference,  Vancouver,  Canada,  March  2012. 

80  Chen,  W.,  Allen,  J.K.,  Mavris,  D.N.,  and  Mistree,  F.,  "A  Concept  Exploration  Method  for  Determining  Robust  Top- 
Level  Specifications"  (1997),  Engineering  Optimization,  Vol.  26,  pp.  137-158. 

81  Ross,  A.M.  and  Hastings,  D.E.,  "Assessing  Changeability  in  Aerospace  Systems  Architecting  and  Design  Using 
Dynamic  Multi-Attribute  Tradespace  Exploration,"  AIAA  Space  2006,  San  Jose,  CA,  September  2006. 

82  Fitzgerald,  M.E.  and  Ross,  A.M.,  "Sustaining  Lifecycle  Value:  Valuable  Changeability  Analysis  with  Era  Simulation," 
6th  Annual  IEEE  Systems  Conference,  Vancouver,  Canada,  March  2012. 
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while  not  being  so  large  as  to  overwhelm  our  ability  to  effectively  evaluate  (with  minimal  bias) 
that  space.  Both  the  optimization83  and  TSE  communities  have  faced  this  problem  and  have 
proposed  a  number  of  techniques  for  managing  the  "problem  of  size"  including  filtering/paring 
the  design  space  through  expert  rules  and  abstraction84,  varying  fidelity  of  the  evaluation  models, 
as  well  as  the  development  of  surrogate  models,  and  aggregation  techniques  on  the  results, 
among  other  techniques85. 
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Figure  9  Techniques  for  building  approximations  in  (Koch,  Simpson,  et  ai  1999,  p.  276) 


Emerging  State  of  the  Art 

Desire  for  a  "new"  TSE-based  approach  within  DoD 

There  is  increasing  recognition  within  DoD  of  the  need  to  increase  knowledge  earlier  in  the 
system  lifecycle86.  An  underlying  premise  in  making  informed  decisions  is  the  concept  that  the 
quality  of  decisions  we  make  is  directly  proportional  to  the  relevant  information  and  knowledge 
we  have  available  at  the  time  of  the  decision.  A  means  to  pull  the  information  content  we  need 
to  make  high-value,  high-confidence  decisions  ahead  of  the  decision-making  event  itself  would 
address  many  of  the  problems  and  challenges  of  early  decisions.  In  Figure  10,  the  left-hand  side 
you  see  the  typical  program  as  it  relates  to  the  degree  of  management  leverage,  cost  committed, 
knowledge,  and  cost  incurred  on  a  program.  Notice  that  the  available  information  (or 
knowledge)  early  in  the  effort  significantly  lags  early  management  decisions  concerning  program 


83  Koch,  P.N.,  Simson,  T.W.,  Allen,  J.K.,  and  Mistree,  F.  (1999),  "Statistical  Approximations  for  Multidisciplinary  Design 
Optimization:  The  Problem  of  Size,"  Journal  of  Aircraft,  Vol.  36,  No.  1,  pp.  275-286. 

84  Murray,  B.,  et  al.,  META  II  Complex  Systems  Design  and  Analysis  (CODA),  United  Technologies  Corporation, 
United  Technologies  Research  Center,  AFRL-RZ-WP-TR-2011-2102,  August  2011. 

85  Gries,  M.,  "Methods  for  evaluating  and  covering  the  design  space  during  early  design  development"  INTEGRATION, 
the  VLSI  journal  38  (2004)  131-183. 

86  Cropsey,  L.,  Informal  correspondence  with  MIT  researchers,  2013. 
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design  architecture,  scope,  and  specification.  Perhaps  just  as  importantly,  at  no  point  in  the 
typical  program  does  the  knowledge  needed  for  decisions  arrive  ahead  of  the  cost  commitment 
curve.  The  implication  is  that  we  consistently  make  decisions  that  significantly  influence  the  cost 
of  procurements  without  the  knowledge  needed  to  make  them.  The  final  point  to  make  is  that 
our  information  or  knowledge  in  a  typical  program  is  usually  a  function  of  the  incurred  costs— 
the  more  we  spend  on  the  program,  the  more  information  we  have  available  (although  under 
the  current  approach,  this  usually  means  we  have  correspondingly  less  leverage  over  future 
program  direction).  While  this  is  perhaps  a  logical  series  of  connections,  it  leaves  us  in  the 
unenviable  position  of  having  to  make  decisions  without  the  information  needed  to  create  high- 
value,  high-confidence  decisions. 


Classic  paradigm 


New  paradigm 


Figure  10  Changing  the  classical  SE  paradigm  with  earlier  knowledge  (adapted  from  Fabrycky  and  Blanchard 

1991) 


The  single  most  effective  way  to  pull  knowledge  ahead  of  decisions  is  to  thoroughly  explore  the 
boundaries  of  the  available  tradespace  at  a  low  level  of  fidelity  before  selecting  a  specific  set  of 
alternative  architectures  to  develop  in  more  depth.  Rather  than  attempting  to  drive  out 
uncertainty  and  risk  early  in  the  materiel  solution  development  process,  these  unknowns  should 
be  thoroughly  explored  in  a  way  that  facilitates  a  robust  dialogue  between  the  effect  owner  and 
the  system  provider  to  arrive  at  a  common,  shared  understanding  of  the  value  of  various  design 
space  considerations  as  they  relate  to  the  value  attributes  of  the  need  space. 


The  essence  behind  a  true  tradespace  exploration  effort  is  the  mental  switch  from  thinking  about 
specific  design  alternatives  and  their  relative  merits  to  instead  decomposing  the  problem  into 
desired  attributes  of  a  solution  and  the  corresponding  design  vector  that  most  impacts  the 
delivery  of  those  attributes.  In  this  way,  the  number  of  potential  solutions  goes  from  just  a  few 
point  designs  to  literally  hundreds  and  even  thousands  of  potential  alternative  solutions. 


Government  leaders  have  expressed  the  importance  of  "moving  the  knowledge  curve  up".  The 
thrust  of  the  effort  is  to  generate  knowledge  and  insight  into  the  system  before  needing  to  make 
decisions  on  significant  life  cycle  cost  and  performance  objectives.  Tradespace  exploration 
provides  an  effective  way  to  do  that  by  generating  the  broad,  first-order  effects  between  system 
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design  parameters  at  a  relatively  low-level  of  fidelity  (and  low  cost  in  terms  of  both  time  and 
money)  rather  than  choosing  a  few  design  points  to  model  at  high-fidelity  (requiring  more  time 
and  a  lot  of  money).  At  the  early  stages  of  program  definition,  the  insight  available  from  exploring 
these  low-fidelity,  first-order  relationships  is  much  more  important  than  making  high-fidelity 
models  of  individual  design  points. 

AFMC  has  expressed  interest  in  pursuing  a  tradespace-based  approach  for  generating  the  earlier 
knowledge  as  described  above.  In  particular,  they  have  worked  with  AFIT  in  performing  a 
tradespace  study  on  the  trainer-x  program.  The  research  team  could  not  find  publically  available 
information  on  this  study  beyond  informal  communication  with  the  people  involved.  The  interest 
in  the  work  extends  beyond  cost  versus  capability  to  include  utility  as  well.  The  inclusion  of  utility 
would  serve  to  help  with  aggregating  across  multiple  dimensions  of  capability,  as  well  as  to  serve 
as  a  mechanism  for  exploring  sensitivities  "in  the  requirements"  (i.e.  the  espoused  utility 
functions  on  the  capability  attributes).  There  was  additional  interest  in  involving  the  AF/A9 
organization  in  incorporating  TSE-based  approaches  as  they  are  responsible  for  integrating  and 
coordinating  studies  across  AF. 

According  to  a  memorandum  from  the  Department  of  the  Air  Force,  entitled  "Implementation  of 
Contractual  Requirements  Sufficiency,"  dated  November  16,  2012: 

Presentation  of  Life  Cycle  Cost  versus  Capability  Analysis.  AF  requirements  and 
acquisitions  processes  must  be  complimentary  and  aligned  with  fiscal  realities. 
Affordability  discussions  must  take  place  at  all  GO-level  requirements  and 
acquisition  forums.  Presentation  of  life  cycle  cost  versus  capability  tradeoff 
analysis  is  required  for  all  AFROCs,  Air  Force  Requirements  Review  Groups 
(AFRRGs),  Air  force  Review  Boards  (AFRBs),  and  Configuration  Steering  Boards 
(CSBs).,  The  Implementing  Commands  (AFMC  and  AFSPC)  will  support  the 
requirements  sponsor  by  providing  cost  and  capability  analysis  for  all  analysis  of 
alternatives  (AoA)  final  reports,  capability  development  documents  (CDDs),  an 
capability  production  documents  (CPDs).  This  requirement  for  mandatory  use  of 
cost  analysis  is  intended  to  ensure  affordability  is  used  to  inform  decisions 
throughout  a  program's  acquisition  lifecycle. 

This  means  that  the  tradespace  resulting  from  an  AoA  must  demonstrate  "value"  as  the 
relationship  between  "what  you  pay"  (cost)  and  "what  you  get"  (capability,  or  benefit).  At  the 
least  this  requirement  will  force  and  opening  of  the  tradespace  to  consider  tradeoffs  of  what  is 
possible  for  different  budgets,  one  of  the  essential  questions  to  investigate  during  a  TSE  study. 

A  recent  article  by  Kane  and  Bartolomei  makes  the  case  that  tradespace  exploration  and  its 
representation  can  fundamentally  change  acquisition  at  the  Air  Force  (and  DoD  more  generally) 
by  providing  a  boundary  object  for  sharing  knowledge  within  the  acquisition  decision  support 
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system  and  to  provide  top  level  synthesis  of  cost  and  benefit  tradeoffs  for  the  organization87.  A 
top  level  illustration  of  a  notional  tradespace  in  Figure  11  shows  the  essential  concept  within  the 
article,  where  a  situational  awareness  provided  through  a  tradespace  representation  gives 
strategic  insights  for  organization  investments. 


Figure  11  Value  chart  [tradespace]  (benefit  versus  cost)  from  Kane  and  Bartolomei  2013  (p  6) 

According  to  Kane  and  Bartolomei,  the  DoD,  and  the  Air  Force  specifically,  runs  the  risk  of 
insufficient  allocation  of  resources  to  meet  national  needs,  due  in  part  to  information  loss  and 
poor  decisions  at  the  interface  of  the  three  intersecting  decision  support  and  acquisition 
processes  of  matching  budget  (AFCS  or  PPBE),  requirements  (CFLIs  or  JCIDS),  and  acquisition  (AF 
Acq.  or  DAS),  as  seen  in  Figure  12  below. 


87  Kane,  R.,  and  Bartolomei,  J.,  "Taming  the  Tigers:  Recapturing  the  Acquisition  Excellence  of  Our  Planning, 
Programming,  and  Acquisition  Three-Ring  Circus,"  Senior  Leader  Perspective,  Air  and  Space  Power  Journal,  March- 
April  2013,  pp.  4-27. 
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PPBE 


Figure  12  Air  Force  acquisition  decision  support  systems  (left)  mirror  those  of  the  Department  of  Defense  (right) 

from  Kane  and  Bartolomei  p.  8 


Figure  13  illustrates  how  the  tradespace  construct  can  be  used  to  synthesize  information  from 
the  three  parts  of  the  acquisition  decision  support  system  in  the  Air  Force. 


Ongoing  Research  and  Development 

The  academic  work  cited  in  the  "state  of  the  practice"  section  is  a  bit  misleading  in  that 
universities  are  constantly  striving  to  push  the  "state  of  the  art"  and  yet  their  work,  once 
published,  becomes  part  of  the  past.  Each  of  the  universities  discussed  previously  have  continued 
their  work  in  evolving  their  methods,  processes,  and  tools,  especially  with  expanded 
considerations  of  issues  such  as  downstream  (e.g.  manufacturability  and  sustainment)  and 
upstream  challenges  (e.g.  requirements  volatility  and  context  shifts).  For  example,  Penn  State 
ARL  has  expanded  its  tradespace  research  to  incorporate  manufacturability  through  their  work 
with  DARPA's  "instant  foundry  through  adaptive  bits"  project88.  MIT  has  expanded  its  tradespace 


88  http://news.psu.edu/story/147150/2012/08/30/penn-state-arl-lead-defense-manufacturing-research-proiect 
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work  into  applications  to  Systems  of  Systems  (SoS)89,  survivability90,  affordability91,  and  multi¬ 
stakeholder  negotiation92. 

Within  the  Systems  Engineering  Research  Center  (SERC),  Wayne  State  University  is  leading  a  small 
team  of  collaborating  universities,  including  Penn  State  and  GTRI,  in  finding  opportunities  to 
support  NAVSEA  in  conducting  set-based  design  activities  for  ship  design. 


ERS  Program 

Engineered  Resilient  Systems  (ERS)  is  one  of  the  seven  DoD  Science  and  Technology  (S&T) 
Priorities  derived  from  a  comprehensive  analysis  of  recommendations  resulting  from  the 
Quadrennial  Defense  Review  mission  architecture  studies93.  Over  the  past  several  years,  several 
government  briefings  have  shared  the  goals  and  vision  for  ERS.94  95  96  97 

The  ERS  priority  area  initially  consisted  of  five  key  technical  thrusts,  one  of  which  was  Data-Driven 
Tradespace  Exploration  and  Analysis.  Upon  transfer  of  the  ERS  Priority  Steering  Council  (PSC)  lead 
role  from  the  Office  of  the  Secretary  of  Defense  (OSD)  to  the  U.S.  Army  Engineer  Research  and 
Development  Center  (ERDC)  in  late  2012,  the  focus  of  the  tradespace  thrust  shifted  slightly,  from 
supporting  four  other  key  technical  thrusts,  to  being  the  foundation  for  ERS  tradespace  analytics 
and  decision  support96. 

In  a  recent  issue  of  Defense  AT&L  Magazine98,  Mr.  Shaffer  highlights  the  focus  of  ERS  program's 
contribution  to  the  strategic  objective  to  affordably  enable  new  or  extended  capabilities,  as 
follows: 


89  Ricci,  N.,  Fitzgerald,  M.E.,  Ross,  A.M.,  and  Rhodes,  D.H.,  "Architecting  Systems  of  Systems  with  llities:  An  Overview 
of  the  SAI  Method,"  12th  Conference  on  Systems  Engineering  Research,  Redondo  Beach,  CA,  March  2014. 

90  Ross,  A.M.,  Stein,  D.B.,  and  Hastings,  D.E.,  "Multi-Attribute  Tradespace  Exploration  for  Survivability,"  Journal  of 
Spacecraft  and  Rockets,  accessed  April  03,  2014.  doi:  http://arc.aiaa.Org/doi/abs/10.2514/l.A32789 

91  Wu,  M.S.,  Ross,  A.M.,  and  Rhodes,  D.H.,  "Design  for  Affordability  in  Complex  Systems  and  Programs  Using 
Tradespace-Based  Affordability  Analysis,"  12th  Conference  on  Systems  Engineering  Research,  Redondo  Beach,  CA, 
March  2014 

92  Fitzgerald,  M.E.,  and  Ross,  A.M.,  "Controlling  for  Framing  Effects  in  Multi-Stakeholder  Tradespace  Exploration," 
12th  Conference  on  Systems  Engineering  Research,  Redondo  Beach,  CA,  March  2014 

93  SECDEF  Memorandum,  Science  and  Technology  (S&T)  Priorities  for  Fiscal  Years  2013-17  Planning,  April  19,  2011. 
Washington,  DC.  OSD  02073-11 

94  Neches,  R.,  Engineered  Resilient  Systems  (ERS):  Insights  and  Achievements  within  the  ERS  Secretary  of  Defense 
Science  and  Technology  (S&T)  Priority,  October  25,  2012 

95  Holland,  J.P.,  Engineered  Resilient  Systems  (ERS):  A  DoD  Science  and  Technology  Priority  Area,  Overview 
Presentation,  Feb  28,  2013 

96  Holland  J.  P.  ERS  PSC  Lead.  Engineered  Resilient  Systems  (ERS)  -  A  DoD  Science  and  Technology  Priority  Area. 
Overview  Presentation,  Feb.  28,  2013.  Director,  U.S.  Army  Engineer  Research  and  Development  Center  (ERDC). 
Director,  Research  and  Development  U.S.  Army  Corps  of  Engineers. 

97  Goerger,  S.,  Engineered  Resilient  Systems  (ERS)  Overview  for  Model-Based  Enterprise  Summit  2013,  December 
18,  2013 

98  Shaffer,  A.,  Concepts  for  Change:  DoD's  2014  Research  and  Engineering  Strategy,  Defense  AT&L  Magazine, 
January-February  2014 
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Engineered  Resilient  Systems  (ERS):  Our  focus  in  ERS  is  to  develop  engineering  concepts, 
science,  and  design  tools  to  protect  against  malicious  compromise  of  weapon  systems  and 
to  greatly  enhance  the  manufacturability  of  trusted  and  assured  defense  systems  across 
the  acguisition  life  cycle.  Through  the  ERS  initiative,  the  department  is  developing  an 
integrated  suite  of  computational  modeling  and  simulation  capabilities  and  systems 
engineering  tools,  complete  with  an  open-reference  architecture,  directly  aligned  to 
acguisition  and  operational  business  processes.  The  R&D  investment  in  ERS  focuses  on 
infrastructure,  information,  design  support,  highly  robust  tradespace  analytics,  decision 
support  tools  and  knowledge  environments  to  increase  the  speed  and  efficiency  of  system 
development,  improve  the  effectiveness  of  fielded  systems  and  provide  life-cycle  costs  for 
decision  making.  The  tools  and  procedures  of  ERS  will  produce  more  comprehensive  and 
robust  requirements  suitable  for  many  more  alternative  mission  scenarios  very  early  in  the 
design  process  or  pre-Milestone  A.  The  reuse  of  data  and  models,  distributed  databases 
that  are  searched  jointly,  virtual  collaboration  environments  and  open  architectures  that 
encourage  partnering  will  lead  to  better-informed  acquisition  decisions.  The  engineering 
design  process  is  streamlined,  and  the  manufacturability  of  a  proposed  design  is  explicitly 
investigated  from  both  engineering  and  cost  perspectives  before  design  commitment. 
Finally,  we  are  developing  robust  tools  to  stress  systems  against  new  mission  contexts, 
tactics,  techniques  and  procedures  or  emerging  requirements,  to  permit  precise 
measurement  and  understanding  of  their  impact  on  all  design  and  production  factors. 


ERS  Tradespace  Exploration 

According  to  ERS  Program  overview  briefings,  ERS  technology  anchors  include  "ERS  Framework 
and  Open  Architecture"  and  "Tradespace  Analysis."99  The  purported  benefits  of  tradespace 
analysis  include,  "enables  informed  decisions,"  "empowers  AoA  and  Requirements  Generation," 
and  "'visualizes'  trades  of  many  more  designs  in  far  less  time."  The  implied  benefit  of  the  last 
item  is  the  discovery  of  "better"  solutions  in  terms  of  cost  and  capability  than  would  be  the  case 
in  investigating  fewer  designs.  The  inputs  into  tradespace  analysis  in  the  ERS  vision  include  both 
the  mission  context  and  the  design  elements  to  be  traded,  considered  throughout  the  "product 
lifecycle"  from  "design  to  disposal"  and  which  also  consider  sensitivity  and  confidence  analysis 
of  the  results. 


99  Holland,  J.,  "Engineered  Resilient  Systems  (ERS)",  ERS  overview  briefing,  May  21,  2014 
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Throughout  Product  Lifecycle 

Design  Disposal 


Figure  14  ERS  tradespace  analysis  for  informing  decisions  through  visualizing  trades  of  many  design  in  less  time 

(from  Holland  2014) 


Due  to  its  essential  role  within  ERS,  this  research  project  investigated  tradespace  exploration  in 
context  of  the  ERS  program  and  its  objectives.  Spero  et  al.100  say  "tradespace  exploration  for  ERS 
is  envisioned  to  coalesce  pertinent  information  tuned  to  specific  decision  makers,  at  the 
appropriate  time,  presenting  a  holistic  view  of  decision  impacts  on  required  system  capabilities." 

As  summarized  in  Spero  et  al.102,  "the  ERS  tradespace  is  not  envisioned  simply  to  be  populated 
with  more  design  alternatives  and  additional  metrics  as  compared  to  a  traditional  tradespace;  it 
is  envisioned  to  be  interactive,  to  be  capable  of  being  supplemented  with  new  information  in 
real-time  as  the  design  process  proceeds,  and  to  include  criteria  not  traditionally  used  in  early- 
phase  design-decision  making  because  of  inherent  uncertainty  or  insufficient  knowledge.  The  ERS 
tradespace  is  envisioned  to  be  collaborative,  persistent,  updated,  and  consulted  throughout  the 
system  lifecycle.  It  is  expected  to  be  a  common  interface,  between  multiple  decision  makers  at 
multiple  levels  and  stages  of  the  design  and  development  process,  that  indicates  possible 
alternative  systems,  the  compromises  required  for  achieving  them,  and  the  effects  of  decisions 
made." 


The  ERS  TSE  desired  capabilities,  while  not  exhaustive,  include  the  following  capabilities 
extracted  from  multiple  briefings  and  recent  papers. 

•  Ability  to  readily  identify  both  physical  and  preference  constraints  on  feasible  solutions 


100  Spero,  E.,  Avera,  M.,  Valdez,  P.,  Goerger,  S.,  "Tradespace  Exploration  for  the  Engineering  of  Resilient  Systems," 
Conference  on  Systems  Engineering  Research  (CSER),  March  2014 
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•  Ability  to  link  tradespace  tool  with  combat  simulations(s)  for  inclusion  of  mission  analysis 

•  Ability  to  perform  analysis  across  alternative  futures 

•  Ability  to  assess  policy  robustness  of  alternative  solutions 

•  Auto-analysis  of  design  elements 

•  Comparing  point  designs  to  alternatives  in  a  tradespace 

•  Computation  for  tradespace  calculation  in  SysML 

•  Data  visualization  -  visualizes  models  (graphs,  charts,  3_D  objects) 

•  Differentiation  of  alternatives  in  terms  of  their  ability  to  change  state  (i.e.,  "flexibility") 

•  Exploration  of  alternatives  robust  to  requirements  shifts 

•  Evaluating  sets  of  alternatives  against  preferences  on  outcomes 

•  Performing  portfolio  analysis 

•  Representation  to  illustrate  the  differential  impact  of  uncertainty  across  a  tradespace  of 

•  alternatives 

•  Reusing  data  from  prior  studies 

•  Seeing  the  immediate  effects  of  changing  needs 

•  Sensitivity  &  confidence  analysis 

•  Supports  set-based  design 

•  Tradespace  visualization  of  multiple  parameters 

A  recent  study  (see  Spero  et  al.100)  provides  an  ERS  view  of  tradespace  exploration,  noting  that 
having  a  valid  set  of  attributes,  and  an  understanding  of  how  a  cross-section  of  tools  can  satisfy 
them,  is  presently  insufficient.  The  authors  discuss  the  need  for  a  deeper  understanding  of  how 
these  tools  are  used  and  how  they  can  be  used  when  performing  tradespace  exploration  in 
support  decision  analysis.  The  study  identified  81  candidate  tradespace  exploration  tools  (and 
also  recognized  more  exist).  It  assembled  a  "best  common  practice"  process  for  their 
requirements,  identified  a  set  of  attributes  for  an  ideal  tradespace  exploration  tool,  and  surveyed 
existing  tools  that  satisfy  these  attributes. 

Desired  outcomes,  as  a  result  of  achieving  the  goals  of  ERS  TSE,  are  enumerated  in  various 
briefings  and  publications.  While  not  an  exhaustive  list,  the  following  desired  outcomes  have 
been  articulated. 

•  Accelerate  requirements  analysis  via  surrogate  response  surfaces 

•  Affect  feedback  between  manufacturability  and  capability 

•  Consider  (lOOx)  number  of  parameters  and  scenarios  in  setting  requirements 

•  Capabilities  for  handling  a  broader  range  of  solutions  in  tradespace  analysis 

•  Data  and  lessons  learned  retained  for  rework 

•  Empowers  AoA  and  requirements  generation 

•  Enables  larger  tradespaces,  keep  alternative  design  options  open  longer 

•  Generate  requirements  that  consider  more  alternative  scenarios 

•  Identification  of  risk-reducing  technologies 

•  Integration  of  tradespace  analysis  into  single  architecture 

•  Integration  into  tool  with  Plug-and-Play  Modules  and  Tool  Orchestrator 

•  Make  much  more  informed  decisions  early  in  acquisition  process 
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•  SoS  insights  more  deeply  understood  and  utilized 

•  Supporting  mission  context  analysis 

•  Tradespace  analysis  support  for  cost,  schedule,  and  performance  risk  analysis 

•  Tradespace  analytics  thrust  -  support  collaborative  analyses,  tools  incorporating  risk  and  cost, 
and  user-friendly  control  of  distributed  systems 

•  Visualization  of  quantified  assessment  of  feasibility/affordability 

•  "Visualizes"  trades  of  many  more  designs  in  far  less  time 

There  are  various  enablers  that  have  been  identified  supporting  these  outcomes.  These  include 
ample  data  storage,  multi-physics  based  modeling  design  tools,  ontologies  describing  system  and 
trades,  and  toolsets  linking  mission  context  to  tradespace  tools.  Other  enablers  include  enhanced 
visualization  in  the  toolsets  and  reduced  order  surrogate  models  with  iterative  ability  of  adjusting 
scenarios  and  requirements  to  physical  capability. 


Toward  an  ERSTSE  Research  Agenda 

A  research  agenda  for  ERS  TSE  has  been  emerging  as  a  result  of  several  recent  workshops  and 
ongoing  research  programs  in  universities  and  government.  The  U.S.  Army  Research  Laboratory's 
Vehicle  Technology  Directorate  (VTD),  in  collaboration  with  the  Office  of  the  Deputy  Assistant 
Secretary  of  Defense  for  Systems  Engineering,  held  an  invitation-only  workshop  on  Data-Driven 
Tradespace  Exploration  and  Analysis  on  July  17-18,  2012.  The  output  from  this  workshop  served 
as  input  to  a  workshop  on  tradespace  and  affordability  that  was  hosted  the  following  day  by  the 
SERC,  for  which  a  report  was  published101. 

Attendees  in  these  workshops  view  resilience  in  the  context  of  ERS  as  more  than  robustness; 
resilience  implies  that  when  the  system  is  placed  into  an  environment  in  which  it  was  not 
originally  intended  to  operate,  after  some  degradation  in  performance,  the  system  can  be 
adapted  or  reconfigured  to  perform  at  its  intended  levels.  To  support  design  for  resilience,  more 
alternatives  must  be  generated  earlier,  considered  longer,  explored  over  multiple,  dynamic 
alternative  futures,  and  searched  exhaustively.  Current  TSE  practice  and  toolsets  are  inadequate. 

The  Data-Driven  Tradespace  Exploration  and  Analysis  workshop  was  attended  by  40  academic, 
government,  and  industry  researchers  and  practitioners  involved  in  tradespace  exploration  for  a 
variety  of  engineering  domains.  The  one-and-one-half  day  workshop  sought  to  develop  near  and 
far  term  tradespace  technology  research  recommendations  for  the  ERS  Priority  Steering  Council 
(PSC)  Lead.  It  was  organized  to  discuss  current  methods,  process,  and  tools  for  performing  these 
tradespace  analysis  related  tasks  and  to  better  understand  existing  tradespace  capabilities  and 
their  suitability  for  ERS.  To  determine  promising  research  areas,  workshop  attendees  were  asked 
to  describe  desired  tradespace  capabilities,  the  associated  current  approach  and  its  deficiencies, 
and  gaps  between  the  two  states.  These  research  areas  were  summarized  in  statements  of  need, 
supporting  rationale,  and  investment  timeframe. 


101  Tradespace  &  Affordability  Program  (TAP)  Workshop  Report,  Stevens  Institute  of  Technology,  Washington,  DC. 
July  18-19,  2012.  vl.O 
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The  results  of  the  workshop  have  been  summarized  in  a  publication,  providing  a  foundational 
research  agenda102.  The  research  agenda  was  elaborated  by  break-out  groups  for  five  perceived 
functions  of  TSE: 

1.  Broadening,  Populating,  and  Managing  the  Tradespace 

2.  Linking  the  Tradespace 

3.  Searching,  Exploring,  and  Analyzing  the  Tradespace 

4.  Acting  on  the  Tradespace 

5.  Modeling  and  Simulation's  Role  in  Generating  Tradespace  Data 

Spero  et  al.102  highlight  their  important  assumption  that  "TSE  is  a  collaborative  effort  in  support 
of  the  decision  making  process  throughout  a  system's  lifecycle,  there  is  no  independent  function 
of  "perform  tradespace  exploration",  and  the  responsibility  to  support  decision  making  is  shared 
across  those  who  generate,  store,  search,  and  act  on  data". 


Progress  Toward  an  ERS  Tradespace  Exploration  Lexicon 

A  vocabulary  for  tradespace  exploration  has  informally  been  emerging  in  the  community,  but 
there  has  been  limited  effort  to  establish  a  common  lexicon.  As  the  tradespace  exploration 
community  expands  to  a  diverse  set  of  practitioners  and  is  used  in  discussions  with  various 
stakeholders,  a  lexicon  is  necessary  to  ensure  there  is  no  ambiguity  of  terms.  Further,  the  lexicon 
will  be  important  to  developing  and  transferring  MPTs  across  the  community.  This  research 
gathered  an  initial  set  of  terms,  and  can  serve  as  the  basis  for  evolving  a  standard  lexicon  for  the 
ERS  community. 

The  list  below  show  the  set  of  terms  gathered  during  this  research  project.  It  should  not  be 
considered  as  complete,  nor  does  it  represent  a  community  consensus. 

Preliminary  Lexicon  of  elements/constructs  used  in  tradespace  exploration 

•  Adaptability 

•  Aggregation  functions 

•  Alternative  solutions 

•  Analysis  of  alternatives 

•  Architecture 

•  Attributes 

•  Attribute  limit 

•  Attribute  units 

•  Attribute  weights 

•  Benefits 

•  Characteristics  (required  to  satisfy  performance  standards) 

•  Changing  operational  environments 

•  Code  fidelity 

•  Complexity 

•  Confidence  analysis 

•  Constraints 


102  Spero,  E.  Bloebaum,  C.,  German,  B.,  Pyster,  A.,  Ross,  A.M.,  "A  Research  Agenda  for  Tradespace  Exploration  and 
Analysis  of  Engineered  Resilient  Systems",  Conference  on  Systems  Engineering  Research  (CSER),  March  2014 
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•  Constructed  model 

•  Contexts 

•  Course  of  action 

•  Cost 

•  Cost  model 

•  Cost  v.  capability  analysis 

•  Decisions 

•  Design 

•  Design  elements 

•  Design  vector 

•  Design  variables 

•  Enumeration  (e.g.,  of  design  vector) 

•  Epoch 

•  Epoch  variables 

•  Epoch-Era  Analysis 

•  Era 

•  Evaluation 

•  Evidence 

•  Expense 

•  Exploration 

•  Feasible  region 

•  Flexibility 

•  Generation 

•  High-level  system  design  variables 

•  llities 

•  Infrastructure 

•  Key  intermediate  variables 

•  Lessons  learned 

•  Lifecycle  costs  (based  on  stakeholder  values) 

•  Lifecycle  cost  analysis 

•  Lifecycle  data 

•  Lifecycle  intelligence 

•  Lifecycle  path  analysis 

•  Lifecycle  schedule  analysis 

•  Mapping  of  stakeholder  values  to  design  variables 

•  Mental  model 

•  Metrics  (e.g.,  Filtered  Outdegree,  Pareto  Trace,  Normalized  Pareto  Trace,  Fuzzy  Normalized  Pareto  Trace) 

•  Mission  context 

•  Mission  data 

•  Mission  scenario(s) 

•  MOP,  MOE,  and  other  performance  attributes 

•  Multi-attribute  utility 

•  Multi-concept  tradespace 

•  Multi-dimensional  data  analysis 

•  Ontology 

•  Operational  performance  characteristics 

•  Options 

•  Optimization  study 

•  Optimum  feasible  space 

•  Parameters 

•  Pareto  frontier 

•  Performance  model 

•  Perturbations 
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•  Preferences 

•  Physics-based  models 

•  Point-based  design 

•  Post-analysis  studies 

•  Predictable  degradation 

•  Probabilistic  risk  assessment 

•  Qualitative/quantitative  ranking  of  how  alternatives  meet  measures 

•  Requirements  generation 

•  Resilience 

•  Responses 

•  Robustness 

•  Sampling  strategy 

•  Scenarios 

•  Selection  (rules) 

•  Sensitivity  plots 

•  Set-based  design 

•  Single-attribute  utility 

•  Stakeholder  needs 

•  Stakeholder  values  (breakdown  into  roles,  attributes,  specific  tasks) 

•  Surrogate  models 

•  System  lifecycle 

•  System  parameters 

•  Swing  weights 

•  Test  for  robustness 

•  Tradespace 

•  Tradespace  analysis 

•  Tradespace  database 

•  Tradespace  information 

•  Tradespace  metrics 

•  Tradespace  plot 

•  Tradespace  yield  number 

•  Uncertainty  analysis 

•  Utility-cost  plot 

•  Utility  curves 

•  Validation 

•  Value  measures  (for  given  mission  scenarios  and  requirements) 

•  Value  model 

•  Visualizations 

•  Weightings 


Term 

Definition 

Example 

Attributes 

stakeholder-defined 

criteria  that  reflect  how 

well  stakeholder- 

defined  objectives  are 
met  for  a  system 

Maritime  Security  SoS  Case: 

•  Primary  stakeholder  identified  the  probabilities  of 
detection/identification/interception  as  the  main 
attributes  that  would  generate  value  for  the  SoS. 

•  A  secondary  stakeholder  identified  the  probability  of 
successful  search  and  rescue  as  an  additional  attribute. 
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Phase  2  Research 


In  this  phase  of  the  research,  TSE-related  concepts  were  codified  into  key  elements,  processes, 
and  descriptors  in  order  to  provide  a  framework  for  capturing  knowledge  generated  during  a  TSE 
study,  facilitating  transfer  to  others,  as  well  as  enabling  reuse  and  comparison  within  future 
studies.  This  phase  was  accomplished  through  coding  TSE  constructs,  and  testing  this  on  a  limited 
basis  through  an  exploratory  pilot  case.  Additionally,  this  phase  investigated  current  the  ERS 
architecture  and  artifacts  in  relation  to  knowledge  capture  and  transfer  of  the  identified  core  TSE 
constructs.  Preliminary  observations  and  next  steps  are  provided  for  evolving  the  ERS  TSE 
architecture  toward  achieving  its  vision  as  a  key  decision  support  capability  for  enabling  resilient 
development  processes  and  systems. 


Framing  Types  of  Knowledge  for  Capture 


When  considering  the  types  of  knowledge  to  be  captured  for  TSE,  it  is  important  to  identify  the 
core  constructs  at  the  essence  of  a  TSE  study  for  decision  support.  Figure  15  below  illustrates  the 
highest  level  abstraction  for  these  core  constructs. 


Decision 

(Problem) 


Decision 

(Solution) 


Figure  15  Key  models  and  spaces  in  the  "design  loop"  for  decision  support  in  TSE  (based  on  Ricci  et  ai  2014103 

and  Ricci  2014104) 


The  "design  loop"  depicted  in  the  figure  above  essentially  represents  the  key  elements  needed 
for  tradespace  exploration  for  decision  support.  The  design  space  represents  the  space  of 


103  Ricci,  N.,  Schaffner,  M.,  Ross,  A.M.,  Rhodes,  D.H.,  and  Fitzgerald,  M.E.,  "Exploring  Stakeholder  Value  Models  via 
Interactive  Visualization,"  in  Conference  on  Systems  Engineering  Research,  Redondo  Beach,  CA,  March  2014. 

104  Ricci,  N.,  Dynamic  System  Perspective  on  Design:  Ility-Driving  Elements  as  Responses  to  Uncertainty,  Dual  Master 
of  Science  Thesis,  Aeronautics  and  Astronautics  and  Technology  and  Policy  Program,  MIT,  June  2014. 
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possible  solutions;  the  performance  space  represents  the  design  space  evaluated  in  terms  of 
performance  criteria  of  interest;  the  value  space  represents  the  performance  space  valuated  in 
terms  of  value  metrics  of  interest.  Models,  both  performance  and  value,  are  used  for  the 
transformations  between  the  spaces  as  indicated.  Models  are  essential  tools  in  system  design 
and  are  used  by  analysts  and  engineers  throughout  the  design  process.  A  key  function  of  models 
is  to  generate  data  when  empirical  sources  are  not  available;  this  data  is  then  used  by  analysts 
and  decision  makers  to  discern  among  potential  alternatives.  Against  the  backdrop  of  spaces  and 
models  are  supplied  inputs  including  needs  (e.g.  mission  objectives,  requirements),  constraints 
(e.g.  workforce  rules,  regulations,  inherited  technology,  etc.),  and  contexts/scenarios  (e.g. 
operational  environments,  regulatory  regimes,  budgetary  conditions,  etc.).  In  addition  to  these 
factors,  two  additional  ones  must  be  considered,  which  relate  fundamentally  to  the  dynamic 
uncertainties  faced  by  the  system.  Perturbations  describe  anticipated  imposed  changes  on  any 
of  the  inputs,  models,  or  spaces.  Responses/options  describe  potential  system  design  and/or 
operational  choices  that  can  be  used  to  mitigate  downside  impact  (i.e.  risk)  or  take  advantage  of 
upside  impact  (i.e.  opportunities)  of  the  perturbations.  The  "design  loop"  is  followed  from 
problem  formulation  through  the  spaces  and  model  transformations  until  sufficient  evidence  is 
generated  as  to  justify  a  solution  (or  set  of  solutions).  Updating  the  problem  formulation,  inputs, 
space  definitions,  and  models  can  all  occur  during  TSE  activities. 


Construct 

Role 

Decision  (problem) 

The  particular  top  level  formulation  of  what  needs  to  be  "solved"  by  a  system 
development  activity 

Decision  (solution) 

The  particular  set  of  answers  to  the  decision  problem,  possibly  including  one 
or  more  proposed  system  solutions  with  appropriate  supporting  evidence 

(Perceived)  Needs 

The  espoused  objectives  (typically  mission-related),  goals,  criteria  of  desired 
capabilities;  can  vary  across  individuals/organizations/time;  sometimes 
represented  as  requirements 

Constraints 

Imposed  limitations  on  system  design,  operation,  resources,  etc. 

Contexts/scenarios 

The  particular  exogenous  factors  that  affect  perceived  system  success, 
including  operating  environment,  regulatory  regime,  budgetary  conditions, 
etc. 

Design  Space 

The  span  of  potential  system  designs  from  which  specific  alternatives  will  be 
evaluated 

Performance  Space 

The  span  of  performance  evaluated  system  alternatives;  the  definition  of  this 
space  is  determined  by  which  performance  metrics  are  of  interest,  these  can 
include  "cost"  as  well  as  behavior 

Value  Space 

The  span  of  value  provided  by  system  alternatives  via  value  model 
interpretation  of  the  performance  space;  the  span  of  this  space  illustrates 
what  is  "possible"  in  terms  of  benefits  and  costs  and  is  subjectively 

Performance  model 

A  model  used  to  predict  the  performance  (e.g.  behavior,  cost)  of  a  system;  a 
formal  representation  of  how  design  alternatives  perform  subject  to 
contexts  and  constraints 

Value  model 

A  model  used  to  predict  the  value  (e.g.  goodness)  of  a  system;  formal 
representation  of  how  performance  criteria  relate  to  perceived  value  (i.e. 
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benefit  at  cost),  subject  to  perceived  needs.  Note:  value-models  attempt  to 
codify  perception  and  therefore  can  and  will  vary  across 
individuals/organizations/time 

Perturbations 

Imposed  changes  to  any  of  the  inputs,  models,  or  spaces  (E.g.  requirements 
change) 

Responses/Options 

The  design  and/or  operational  response  of  the  system  in  order  to  mitigate 
the  downside  consequences  of  perturbation(s);  this  is  what  enables 
resilience  in  systems  (i.e.  robustness,  versatility,  adaptability/flexibility, 
survivability,  etc.) 

In  addition  to  these  tradespace  elements,  the  information  about  the  study  itself,  as  well  as  the 
process  followed,  should  be  captured.  The  study  description  at  a  top  level  should  include  an 
overview,  identification  of  key  decision  makers,  the  goals  for  the  study  (including  description  of 
"problem  to  be  solved"  and  criteria  for  determining  when  a  solution  is  in  hand),  approach  for  the 
study,  findings  of  the  study  (e.g.  the  proposed  solution),  and  outcome  (i.e.  impact/results  of 
using/deploying  the  findings).  The  process  description  should  be  standardized,  such  as  the 
generic  tradespace  exploration  set  of  processes  described  earlier  in  this  report  under  "Scope: 
Tradespace  Exploration"  and  repeated  here:  generation,  enumeration,  sampling,  evaluation, 
exploration,  and  selection.  Use  of  a  standard  description  facilitates  knowledge  capture  and 
transfer  across  TSE  studies. 


Exploration  Case  Study 

One  of  the  tasks  in  this  research  project  was  to  conduct  an  exploratory  case  study  to  identify 
tradespace  exploration  artifacts  in  practice  and  how  they  relate  to  knowledge  goals  within  ERS. 
Unfortunately  delays  in  data  availability  cut  short  the  intended  timeframe  across  which  to 
perform  this  case  study.  What  follows  are  some  observations  based  on  limited  data  availability 
to  two  Navy  tradespace  exploration  activities  conducted  in  concert  with  the  ERS  Program. 


Navy  TSE  Study  1:  Point  vs.  Set-based  Design 

Overview:  In  this  study,  a  team  at  Naval  Surface  Warfare  Center-Carderock  conducted  a  ship 
design  study  using  an  "ERS"  approach.  The  vision  included  "solid  'framing  assumptions'  in  pre¬ 
milestone  A  efforts"  in  order  to  manage  the  fact  that  "early  decisions  drive  significant  and 
expensive  results."  Specifically  ERS  would  enable  these  improved  decisions  through  "physics 
based  data-driven  trade  space  exploration,"  and  "robust  analysis  of  requirements,  design 
concepts,  CONOPs,  mission  effectiveness,  technology,  and  cost".105 

Key  decision  makers:  Navy  internal,  ERS  Program 

Goal(s)  for  the  study:  "Demonstrate  ability  to  design  a  resilient  ship,  with  a  resilient  process, 
through  application  of  physics  based  modeling  &  trade  space  informed  set-based  design" 


105  Mackenna,  A.,  and  Hough,  J.,  "Engineered  Resilient  Systems  (ERS)  for  Ship  Design  &  Acquisition,"  Distro  A  briefing, 
May  21,  2014,  slide  6. 
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Approach:  start  with  baseline  design;  two  independent  teams  to  develop  final  ship  design; 
perturbations  introduced  during  design  and  after  initial  "final  design"  (e.g.  midlife  upgrade 
scenario)  to  test  resiliency  of  design  process  and  of  final  design 

Tradespace  elements  description: 


Type 

Element 

Description 

Inputs 

Needs 

unknown 

Constraints 

unknown 

Contexts/Scenarios 

early  stage  design  phase,  service  life  phase 

Spaces 

Design  Space 

inherited  baseline  design;  set-based  team  used  Rapid  Ship 

Design  Environment  (RSDE)  to  expand  space 

Performance  Space 

unknown 

Value  Space 

measure  of  effectiveness,  cost,  and  risk 

Models 

Performance  Model 

ASSET-LEAPS  (includes  semi-empirical  and  physics-based 
analysis  tools  and  performance-based  cost  model) 

Value  Model 

MS-Excel  spreadsheets  for  MOE  and  risk  assessment 

Changes 

Perturbations 

requirements  shift  during  design  phase  and  service  phase 

Responses/Options 

to  requirement  shift  in  design  phase:  point-based  had  to 
redesign,  while  set-based  just  picked  different  design;  to 
requirement  shift  during  service  phase:  unknown 

Tradespace  processes  description: 


Process  Description 

Generation 

begin  with  baseline  design 

Enumeration 

point-based  team:  propose  few  variants  off  of  evolved  baseline;  set-based  team:  use 

RSDE  leveraging  multi-discipline  optimization  to  propose  alternatives 

Sampling 

point-based  team  likely  uses  full  enumerated  (small)  set  as  sample;  set-based  team 
appears  to  have  leveraged  RSDE's  optimization  capabilities  to  derive  samples  from  full 
potential  enumeration 

Evaluation 

Both  teams  used  ASSET-LEAPS  to  evaluate  performance  and  cost  of  sampled  design 
points,  with  value  evaluated  through  a  MS-Excel  spreadsheet  model;  surrogate  modeling 
is  available  through  RSDE  as  well  (including  Kriging,  among  others) 

Exploration 

unknown  for  point-based  team;  set-based  team  uses  RSDE  GUI  to  generate  visualizations 
of  design  space  for  exploration 

Selection 

each  team  proposed  a  "final  design"  at  the  end  of  each  phase  to  be  judged  by  highest 

MOE  and  lowest  cost  (and  risk) 

Infrastructure  used:  ASSET-LEAPS  and  RSDE  tools 


Key  findings:  When  faced  with  design  phase  requirements  change,  point-based  design  team  had 
to  conduct  rework,  while  set-based  design  team  did  not;  when  trying  to  meet  cost  goals,  point- 
based  team  had  to  "guess  on  way  to  achieve  cost  goal  based  on  experience  of  team  members," 
whereas  set-based  team  "gained  knowledge  of  design  &  cost  drivers"  and  therefore  "had 
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knowledge  on  how  to  meet  cost  goal"  without  having  to  guess  or  rely  upon  team  experience; 
point-based  team  tended  to  pick  higher  risk  designs  than  set-based  team  as  requirements 
changed 

Outcome:  Pilot  study  validated  ERS  hypothesis  through  results  of  point  vs  set-based  design 
processes  in  response  to  requirements  change  perturbation  (i.e.  tradespace  "facilitates  rigorous 
requirements  analysis"  allowing  "deliberate  cost  vs.  capability  decisions  at  earliest  stages  of 
design  acquisition  (pre-milestone  A)."  Additionally,  it  was  found  that  using  physics  based  analysis 
tools  enabled  early  identification  of  unobtainable  or  unaffordable  requirements  (i.e.  the 
infeasible  regions  of  the  evaluated  tradespace).  It  also  became  apparent  that  the  knowledge 
embedded  in  the  structure  of  the  tradespace  mimicked  that  of  experienced  members  and  could 
be  used  to  educate  inexperienced  ship  designers,  potentially  serving  as  a  knowledge  transfer 
mechanism  when  the  experienced  team  members  are  unavailable.  The  vision  for  ERS  Ship  Design 
includes  expanded  sets  of  physics-based  modeling,  and  more  formal  representations  for  decision 
support  (i.e.  showing  requirements  tradeoffs,  MOE,  cost  versus  requirements,  risk,  etc.) 


Navy  TSE  Study  2:  AoA  Extension 

Overview:  In  this  study,  a  formal  AoA  (~9  mos  level  of  effort)  was  conducted,  resulting  in 
approximately  5  point  designs.  The  results  of  the  study  were  not  conclusive,  with  ships  deemed 
to  be  too  expensive  and  not  capable  enough.  In  order  to  demonstrate  the  potential  benefit  of 
moving  beyond  a  traditional  AoA  and  into  a  cost  versus  capability  analysis  (i.e.  a  TSE  study),  the 
ERS  program  extended  the  study  by  3  months  (~6-8  person  months  of  additional  effort),  from  1 
Oct  2013  to  15  Jan  2014.  Insights  gained  in  the  tradespace  activity  resulted  in  Navy  directing  next 
effort  to  conduct  cost  versus  capability  analysis  rather  than  a  formal  AoA. 

Key  decision  makers:  Navy-internal  (unknown) 

Goal(s)  for  the  study:  Unknown 

Approach:  Generate  additional  design  alternatives  from  the  baseline  designs  and  evaluate  them 
using  models  in  support  of  a  cost  versus  capability  analysis 

Tradespace  elements  description: 


Type 

Element 

Description 

Inputs 

Needs 

unknown 

Constraints 

unknown,  but  included  upfront  cost  limit 

Contexts/Scenarios 

unknown 

Spaces 

Design  Space 

TBD 

Performance  Space 

TBD 

Value  Space 

TBD 

Models 

Performance  Model 

TBD 

Value  Model 

TBD 

Changes 

Perturbations 

TBD 

Responses/Options 

TBD 
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Tradespace  processes  description: 


Process  Description 

Generation 

unknown 

Enumeration 

traditional  AoA  results  in  ~5  designs;  unknown  how  larger  set  is  enumerated  in  cost  v. 
capability  analysis  extension 

Sampling 

unknown  (full?) 

Evaluation 

unknown 

Exploration 

unknown 

Selection 

unknown 

Infrastructure  used:  unknown 

Key  findings:  initial  AoA  resulted  in  5  point  designs  with  insufficient  capability  and  high  cost;  the 
Cost  v.  Capability  TSE  extension  resulted  in  capability  versus  cost  correlation  insights  (i.e.  design 
drivers) 

Outcome:  Navy  liked  the  approach  enough  to  direct  the  next  effort  (1  Apr  2014  to  15  Aug  2014) 
to  conduct  a  Cost  v.  Capability  analysis  rather  than  a  formal  AoA 


Reflection  on  exploration  cases 

Emergent  in  the  application  of  the  TSE  descriptors  to  the  exploration  case  was  the  realization 
that  the  data  needed  should  be  readily  available  to  the  team,  but  difficult  to  capture  by  an 
outsider.  This  may  be  obvious  after  the  fact,  but  it  means  that  any  proposed  framework  should 
be  readily  available  to  the  TSE  team  prior  to  and  during  a  study,  both  to  assist  with  study  conduct 
and  to  help  with  knowledge  capture.  Not  obvious  in  the  exploratory  case  is  the  fact  that  in  spite 
of  the  multitude  of  "unknown"  responses,  each  item  is  directly  relevant  to  any  TSE  study  and 
should  be  readily  answerable  by  the  TSE  team.  Additionally,  the  item  "infrastructure  used"  was 
not  originally  proposed  as  an  item,  but  emerged  as  a  useful  item  for  capturing  aspects  like  "tools 
used"  or  other  supplied  items  that  are  potentially  useful  beyond  this  particular  study.  Once  broad 
knowledge  capture  is  underway  using  the  ERS  architecture,  such  information  could  become  very 
valuable  in  tracking  "popular"  tools  and  computing  capabilities,  as  well  as  to  provide  some  insight 
into  future  infrastructure  planning  (e.g.  unused  policy  documents  or  oversubscribed  HPC  clusters, 
etc.) 


Framework  for  a  Larger-scale  Case  Study 

Based  on  a  synthesis  of  the  literature,  interviews,  and  the  exploratory  case,  the  following  is  a 
preliminary  template  for  a  description  of  an  ERS-relevant  tradespace  study. 

Overview:  TBD 

Key  decision  makers:  TBD 
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Goal(s)  for  the  study:  TBD 
Approach:  TBD 

Tradespace  elements  description: 


Type 

Element 

Description 

Inputs 

Needs 

TBD 

Constraints 

TBD 

Contexts/Scenarios 

TBD 

Spaces 

Design  Space 

TBD 

Performance  Space 

TBD 

Value  Space 

TBD 

Models 

Performance  Model 

TBD 

Value  Model 

TBD 

Changes 

Perturbations 

TBD 

Responses/Options 

TBD 

Tradespace  processes  description: 


Process 

Description 

Generation 

TBD 

Enumeration 

TBD 

Sampling 

TBD 

Evaluation 

TBD 

Exploration 

TBD 

Selection 

TBD 

Infrastructure  used:  TBD 


Key  findings:  TBD 
Outcome:  TBD 

A  larger  scale  study  would  aim  to  populate  this  template  to  the  next  level  of  detail,  and  should 
also  be  applied  across  additional  cases.  Most  importantly,  a  catalogue  of  responses  to  each  of 
these  descriptors  can  be  used  to  inform  the  ERS  architecture  and  knowledge  repositories  to 
facilitate  future  ERS  TSE  studies  and  tools.  Additionally,  further  enhancement  of  the  descriptor 
set  can  be  used  to  guide  practitioners  to  explicitly  consider  essential  elements  necessary  to 
determine  resilience  of  their  systems  within  their  studies. 


Preliminary  Prescription  to  the  ERS  Program  on  TSE 

In  spite  of  the  limited  availability  of  data  for  the  exploratory  case  study  in  this  research,  enough 
insights  were  gained  to  support  preliminary  prescription  for  ERS  going  forward.  These  are 
described  in  the  following  sections. 
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Assessment  of  "today" 


The  journey  of  the  ERS  program  in  meeting  its  ambitious  goals  is  not  a  short  one  and  requires 
many  milestone  accomplishments  along  the  way.  For  example  see  Figure  16  below  for  the 
architecture  roadmap.  The  roadmap  illustrates  the  rollout  of  key  components  of  the  ERS 
infrastructure  for  supporting  TSE. 


Release  1 


Mission  Context 
Analysis 

System  performance 
within  a  simulated  mission 


Conceptual 

Modeling 

■Ontologies  describe 
system  and  trades 
■Computation  for 
tradespace 
calculation  in  SysML 


Apply 

Leverage 


Build 


Establish  hosting 
environment  for  Navy 


Model  Integration 

Method  to  integrate 
models:  1)  simplification  to 
meta-models,  2)  meta¬ 
models  packaged, hosted 
on  ERS  environment 


Needs  Analysis 

Conducted  with  the  Navy, 
Air  Force  and  Marine  Corp 


CRES-GV  (HPCMO) 
High-fidelity 

design/analysis  tools  for 
ground  vehicles 


ERS  Capability 
Analysis 
■LEAPS  (USN) 
■FACT  (USMC) 

•Da  Vinci 
■Kestrel 
•  WSTAT 
•Ere. 


Establish  ERS 
computing  enclave 


PilotKM 

functions 


Advance  Visualization 
(HPCMO) 

ParaView  provides  insights 
into  myriad  data  generated 
in  analyses 


Manufacturability 
(solicitation) 
Analytics  to  assess 
manufacturability 


Risk  and  Uncertainty 
Measure  and  display  risk 
associated  with  designs 


Navy  applies 
ontology  modeler  to 
LEAPS 


;  YU 


(HPCMO) 

Non-HPC  machines 
interoperable  with 
HPC  resources 


Future 

Releases 

FY16+ 


CREATE  (HPCMO) 
Deploy  advanced 
computational  design 
tools: 

CREATE  -  Air  Vehicles,  - 
Ships,  -Antennas,  - 
Meshing  and  Geometry 


M&S  (DoD) 
Suite  of  DoD  modeling 
and  simulation 
capabilities 


Environmental  Simulator  (EnvSim) 
(ERDC) 

Physics-based  codes  to  create  a  digital  representation 
of  an  environment 


Figure  16  ERS  architecture  roadmap  (from  Holland,  May  2014) 


The  ERS  program  has  made  excellent  progress  in  both  developing  an  infrastructure  vision  with 
capabilities,  and  conducting  pilot  studies  to  validate  the  intended  benefit  of  using  an  ERS 
perspective.  The  ERS  contributions  are  based  partly  on  employing  the  following: 


•  HPC  and  physics-based  models  to  provide  broad  and  deep  data  evidence 

•  Surrogate  modeling  techniques  to  enable  broader  design  space  exploration 

•  Multi-discipline  optimization  for  enumeration  and  sampling  of  design  space 

•  Improved  visualizations  of  cost  versus  capability  and  other  metrics  for  supporting  decision 
making 

•  An  initial  investigation  of  perturbation-response  on  system  design  process  as  well  as  system 
design 

•  Plan  to  provide  infrastructure  to  the  community,  including  standardized  tools,  databases,  and 
processes 


Based  on  the  knowledge  capture  template  in  the  previous  section,  moving  forward  ERS  should 
make  explicit  effort  to  also  incorporate  the  following: 
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•  Coupling  of  scenarios/context  evaluation  with  the  tradespace  exploration  activities 

•  Explicit  "value  model"  use/leverage 

•  Explicit  consideration  of  a  range  of  perturbations,  representing  dynamic  uncertainties  (risks 
and  opportunities)  to  both  development  processes  and  systems  under  consideration  in  a  TSE 

•  Explicit  consideration  of  a  variety  of  responses/options  of  both  development  processes  and 
systems  to  the  perturbations  in  order  to  evaluate  their  resilience 

•  Explicit  consideration  of  operations  trades  in  addition  to  physical  system  trades 


Important  Augmentations  to  T raditional  T radespace  Analysis 

Since  the  purported  goal  of  tradespace  exploration  is  to  support  decisions  (at  least  as 
represented  in  the  ERS  Program  overview  and  within  this  document),  the  framing  of  each  TSE 
study  should  be  positioned  to  most  easily  assist  in  that  capacity.  This  means  the  input  factors 
should  be  "decision  variables"  (i.e.  factors  within  the  control  and/or  influence  of  the  decision 
maker),  and  the  output  factors  should  be  the  key  decision  criteria  to  be  used  by  the  decision 
maker.  In  its  most  general  guise,  this  is  similar  to  an  applied  decision  analysis  exercise,  with  the 
added  complexity  that  the  study  can  generate  additional  alternatives  (i.e.  an  open-ended  set  for 
evaluation).  A  "value-driven"  perspective  has  been  adopted  since  it  most  directly  reflects  this 
decision-oriented  perspective.  Values  are  simply  the  top  level  benefit  and  cost  perceptions  of  a 
given  decision  maker,  including  possible  selection  rules  (i.e.  I  care  about  the  performance  of  my 
system  in  terms  of  spatial  resolution,  tracking  latency,  and  reliability,  as  well  as  O&M  costs  and 
political  palatability  in  terms  of  manufacturing  and  sustainment).  By  focusing  on  values  (i.e.  the 
decision  criteria),  this  opens  up  the  design  space  to  include  vastly  different  alternatives  and 
therefore  increases  the  likelihood  of  finding  a  feasible,  or  possibly  better,  solution.  The  value- 
driven  perspective  is  in  contrast  to  the  alternatives-focused  thinking,  which  starts  with  a 
solution106.  The  value-driven  perspective  also  frees  up  engineers  to  focus  on  engineering  and 
decision  makers  to  focus  on  making  decisions,  rather  than  the  other  way  around. 

Alternatives  focused  thinking: 

•  May  result  in  only  small  number  of  (possibly  inappropriate,  or  subpar)  solutions  considered 

•  More  quickly  reduces  ambiguity  in  the  problem  by  quickly  getting  to  the  concrete  evaluation 
and  specification  part  of  design 

•  May  result  in  using  scarce  resources  developing  solutions  that  need  to  be  justified  (e.g.,  "sold" 
through  altering  expectations) 

Value  focused  thinking: 

•  Allows  for  "re-casting"  of  a  "problem"  into  an  "opportunity" 

•  Increases  likelihood  of  solution  performing  well  in  value 

•  Aligns  scarce  resources  on  the  proper  questions 

•  Allows  for  consideration  of  new  solutions  (helps  to  break  anchoring) 


106  Keeney,  R.L.,  "Value-Focused  Thinking:  A  Path  to  Creative  Decisionmaking,"  Harvard  University  Press,  Cambridge, 
MA,  1992. 
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Some  engineers  do  not  like  the  use  of  "subjective"  factors  for  evaluating  their  designs.  The  fact 
of  the  matter  is  that  good  decision  makers  use  all  necessary  and  relevant  information  for  making 
their  decisions.  Sometimes  these  factors  are  non-technical,  including  issues  of  economic  and 
social  equity  or  even  political.  Good  tradespace  exploration  supports  these  decisions  in  a  layered 
fashion  by  addressing  the  technical  factors  and  leaving  room  for  decision  makers  to  incorporate 
addition  information,  as  needed.  It  is  important  to  also  note  that  subjectivity  enters  into  decision 
making  even  on  technical  factors.  This  subjectivity  comes  about  as  people  interpret  risk  and 
uncertainty,  which  is  inherent  in  any  estimated  performance  or  cost  figures  in  system  studies.  By 
layering  in  the  "subjectivity"  through  tradable  value  models,  the  tradespace  analyst  of  the  future 
will  be  able  to  identify  not  only  the  impact  of  varying  "subjective"  preferences  on  what  makes  a 
"good"  solution,  but  also  be  able  to  identify  potential  solutions  that  are  insensitive  to  changes  in 
those  preferences. 

An  additional  challenge  for  future  tradespace  exploration  and  analysis  is  the  explicit 
consideration  of  the  impact  of  time  and  context  on  the  tradespace  results.  What  may  appear  to 
be  a  good  solution  today  may  change  in  the  future  under  difference  circumstances.  While 
sensitivity  analysis  and  some  scenario  planning  does  occur  today,  it  is  done  insufficiently  to 
identify  truly  resilient  system  solutions.  Sensitivity  analysis  across  changing  contexts  over  time 
should  be  built  explicitly  into  the  ERS  program  architecture  and  associated  MPTs  to  ensure  that 
the  analysis  is  done,  rather  than  relegated  to  the  end  and  likely  to  be  omitted  when  projects  run 
into  time  constraints. 

Additionally,  tradespace  exploration  studies  must  explicitly  consider  the  impact  of  perturbations 
on  the  needs  (e.g.  requirements  change),  constraints  (e.g.  budget  shift),  contexts  (e.g.  new 
mission  environment),  design  space  (e.g.  new  concepts),  performance  space  (e.g.  new 
performance  metric),  value  space  (e.g.  new  benefit  metric),  performance  model  (e.g.  new  physics 
based  model  or  cost  model),  or  value  model  (e.g.  new  utility  function).  Resilience  is  achieved  by 
the  design  process  and  the  system  design  itself  through  their  responses  to  these  perturbations. 
Thus  it  is  essential  that  ERS  TSE  considers  both  perturbations  and  responses  as  necessary 
components  of  any  study.  Otherwise,  resilience  considerations  likely  will  be  ad  hoc  and  less  likely 
to  be  present  by  intent. 


Developing  Further  Enablers 

The  research  investigation  has  emphasized  the  need  for  enablers  for  ERS  tradespace  exploration. 
Infrastructure  in  support  of  computation,  data  storage,  visualization,  and  other  enablers  are 
necessary  to  achieving  the  ERS  TSE  vision.  A  preliminary  lexicon,  included  in  this  report,  needs  to 
be  further  developed.  Having  a  common  set  of  terms  will  support  the  maturation  of  TSE 
processes  and  enable  better  communication  across  the  TSE  research  and  practitioner 
community.  There  are  many  contributors  to,  and  beneficiaries  of,  ERS  TSE.  Further  development 
of  a  "map"  of  the  activities  and  involved  organizations  will  enable  continued  growth  of  a 
community  of  research  and  practice. 


50 


Use  ofTradespace  Exploration  for  Knowledge  Capture 


As  mentioned  earlier,  TSE  has  shown  promise  for  codifying  expert  knowledge  into  a  concise 
format  that  is  accessible  to  novices  and  other  experts  alike107,108.  The  following  sections  revisit 
the  concepts  uncovered  in  this  research  and  their  applicability  for  capturing  and  transferring 
knowledge. 


Revisiting  the  Types  of  Knowledge  for  Capture 

When  proposing  knowledge  capture  in  TSE  activities,  each  of  the  elements  in  Figure  17 
represents  a  body  of  knowledge  both  for  the  specific  study  at  hand  and  the  domain  (or  decision 
process  more  generally). 


Figure  17  Review  of  the  key  study  data,  inputs,  models,  and  spaces  for  decision  support  in  TSE 


For  example,  for  a  given  project  (as  described  by  the  ERS  ontology,  capturing  system  type, 
domain,  etc.),  historical  data  on  similar  projects  should  be  accessible  as  desired  in  order  to  report 


107  Wolf,  D.,  An  Assessment  of  Novice  and  Expert  Users'  Decision-Making  Strategies  during  Visual  Trade  Space 
Exploration,  MS  Thesis,  Mechanical  Engineering,  Penn  State  University,  May  2009. 

108  Even  the  recent  ERS-supported  Navy  Ship  Design  project  found  this  to  be  the  case  when  the  set-based  design 
team  could  leverage  knowledge  in  the  tradespace  revealing  cost  drivers,  whereas  the  point-based  design  team  had 
to  rely  on  expert  knowledge  to  guess  at  how  to  reduce  cost,  resulting  in  the  project  suggesting  that  TSE  would  be 
useful  for  training  novices  (see  Mackenna,  A.,  and  Hough,  J.,  "Engineered  Resilient  Systems  (ERS)  for  Ship  Design  & 
Acquisition,"  Distro  A  briefing,  May  21,  2014) 
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their  relevant  inputs  (e.g.  needs,  contexts,  constraints),  as  well  as  previously  defined  design, 
performance,  and  value  spaces  and  associated  models. 

In  the  space  domain,  ISR  systems  tend  to  have  similar  top  level  KPPs,  which  should  be  readily 
accessible  in  future  ISR  TSE  studies.  Associated  data  with  these  decision  criteria  should  be 
captured  and  reportable  (e.g.  desired  spatial  resolution,  revisit  time,  latency,  etc.)  In  current 
practice  (typically  document  or  process-driven,  rather  than  data-driven),  this  type  of  data  may 
not  be  readily  accessible.  Concerns  relative  to  data  sensitivity  must  be  taken  into  account,  but 
can  be  handled  with  the  appropriate  data  partitioning  and  security  policies. 


Understanding  how  Effort  Impacts  Confidence  in  Knowledge 

In  practice,  not  every  study  has  the  same  amount  of  resources  available  (including  personnel, 
expertise,  computation,  etc.),  and  not  every  study  requires  the  same  level  of  effort  for  the 
decision  problems  posed.  This  leads  naturally  to  an  effort  versus  confidence  (in  knowledge 
gained)  tradeoff  in  terms  of  how  to  conduct  the  TSE  study.  In  the  recent  past,  some  work  was 
done  applying  the  same  TSE  process  at  varying  levels  of  effort  in  order  to  begin  to  understand 
the  inherent  tradeoffs  between  spending  more  time  on  a  particular  activity  (at  the  risk  of  not 
completing  later  activities)  or  less  time  on  a  particular  activity  in  order  to  complete  more 
activities  (at  the  risk  of  not  going  "deep  enough"  in  the  generation  of  the  results).  In  order  to 
address  this  challenge,  the  research  team  developed  a  TSE  method  whose  inherent  structure  led 
to  incremental  knowledge  generation  (i.e.  as  one  progresses  from  earlier  to  later  activities, 
insights  build  upon  prior  ones  so  that  the  analyst  does  not  have  to  wait  until  the  end  to  get 
insights).  This  incremental  TSE  method  is  called  the  Responsive  Systems  Comparison  Method 
(RSC)  and  has  seven  processes,  each  building  upon  the  prior  (see  Figure  18  figure  below)109'110'111. 
For  the  effort  versus  confidence  study,  RSC  was  applied  to  several  different  case  studies  with 
imposed  time  constraints  to  force  effort  tradeoffs.  These  studies  are  more  illustrative  than 
conclusive,  as  more  data  would  need  to  be  generated  for  statistically  valid  prescription  (Figure 
19).  Flowever,  these  studies  are  suggestive  that  it  would  be  valuable  to  conduct  broader 
investigations  into  this  very  practical  issue  of  how  to  allocate  project  resources  when  conducting 
a  TSE  study  within  ERS. 


109  Ross,  A.M.,  McManus,  H.L.,  Long,  A.,  Richards,  M.G.,  Rhodes,  D.H.,  and  Hastings,  D.E.,  "Responsive  Systems 
Comparison  Method:  Case  Study  in  Assessing  Future  Designs  in  the  Presence  of  Change,"  AIAA  Space  2008,  San 
Diego,  CA,  September  2008. 

110  Ross,  A.M.,  McManus,  H.L.,  Rhodes,  D.H.,  Hastings,  D.E.,  and  Long,  A.M.,  "Responsive  Systems  Comparison 
Method:  Dynamic  Insights  into  Designing  a  Satellite  Radar  System,"  AIAA  Space  2009,  Pasadena,  CA,  September 
2009. 

111  Schaffner,  M.A.,  Designing  Systems  for  Many  Possible  Futures:  The  RSC-based  Method  for  Affordable  Concept 
Selection  (RMACS),  with  Multi-Era  Analysis,  Master  of  Science  Thesis,  Aeronautics  and  Astronautics,  MIT,  June  2014 
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Responsive  System  Comparison  method  fidelity  options  and  knowledge  generated 

Study  Name: _  Respondant: _ 

The  following  are  example  options  for  particular  activities  within  RSC  Processes. 

Start  time: _  Goal  end  process: _  Goal  effort  expended: _  (person-hrs) 

The  options  are  listed  from  lowest  to  highest  fidelity  (and  required  effort) 

End  time: _  Actual  end  process: _  Actual  effort  expended: _  (person-hrs) 


RSC  Process 

Pi 

Value-Driving 
Context  Definition 

P2 

Value-Driven 
Desiqn  Formulation 

P3 

Epoch 

Characterization 

P4 

Design  Tradespace 
Evaluation 

P5 

Multi-Epoch 

Analysis 

PS 

Era 

Construction 

P7 

Lifecycle  Path 
Analysis 

Defining  value  network 

Selecting  DM(s) 

Uncertain!!  categories 

S|stem  Model  types 

Select  epochs  for  analysis 

Era  samplig 

list  of  stakeholders 

Use  past  examples/data 

Technology 

Look-up  table  (opinion) 

Set  "choices" 

Set  "choices" 

stakeholders  grouped  by  interest 

Use  proxy  (team  role-playing) 

Policy 

Look-up  table  (data-based) 

Random  sampling 

Random  sampling 

stakeholders  with  value  flows 

Use  proxy  (external  rep.) 

Budget 

Parametric  model(s) 

Partial  factorial  (DOE-based) 

Partial  factorial  (DOE-based) 

Use  real  DM 

Infrastructure 

Simulation(s) 

Full  factorial 

Full  factorial 

Attribute  elicitation 

End  uses 

Design  space  sampling 

Pareto  tracing 

Attributes  wfranges  only 

Epoch  space  enumeration 

Set  "choices" 

Qualitatively  inspect  Pareto  Set 
movement  in  TS  across  epochs 

...  plus  informal  "weights" 

Set  "choices" 

DV-driven  point  sampling(s) 

Calculate  Pareto  Trace 

....  plus  formal  "weights" 

EV-driven  point  enumeration(s) 

Extreme  sampling 

Calculate  Fuzzy  Pareto  Traces 

Utility  elicitation 

Extreme  enumeration 

Random  sampling 

None,  assume  linear 

Partial  factorial  (DOE-based) 

Partial  factorial  (DOE-based) 

Qualitative  "shape"  w/key  values 

Full  factorial 

Full  factorial 

Formal  utility  interview 

Cross-epoch  evaluations 

Changeability  analysis 

Generate  concepts 

inspect  baseline  epoch,  indicate 
sensitivity  (plus.minus  wf 
intensilul 

Use  max(OD)  assessment  using 
transition  rules  (filters  L.M.H) 

Use  past  concepl(s) 

run  "models"  for  each  epoch 

Calculate  FOD  using  matrices 

Use  opinion-driven  concepts 

Use  DVM  to  drive  concepts 

Principle  Knowledge  Generated 

Problem  definition 

Value  proposition(s) 

Key  exogenous  factors 

Design-value  tradeoffs 

Multo-OM  tradeoffs 

Time-based  context  effects 

Value  robust  designs 

Possible  long  run  scenarios 

System  evolution  strategies! 

On  a  scale  from  1-5,  please  assess  your  comfort  with  the  knowledge  generated  by  each  process,  given  the  scope  of  your  study 


1  Not  at  all 

2  A  little 

3  Somewhat 

4  Fairly 

5  Very 

no  results  or  arbitrary  placeholder 

results  likely  to  change 

results  possibly  might  change 

results  unlikely  to  change,  but  may 

results  stable  and  mostly  complete 

results  (should  redo  activity) 

(if  redo  activity) 

(if  redo  activity) 

grow  (if  redo  activity) 

(no  need  to  redo  activity) 

(Please  enter  'O'  if  process  was  not  performed) 


Figure  18  Example  method  scalability  options  with  associate  knowledge 
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6  person-hrs  lOperson-hrs 


Effort 

i 

Problem  definition 

Value  proposition(s) 

Key  exogenous  factors 
Design-value  tradeoffs 
Time-based  context  effects 
Possible  long  run  scenarios 
System  evolution  strategies 

Knowledge 

\ 

Processes 

2 

3 

Comfort  & 

Multi-DM  tradeoffs 

Value  robust  designs 

4 

Completeness 

0 

C 

7 

Figure  19  Example  effort  (completeness)  vs.  knowledge  tradeoffs  in  a  TSE  study  using  RSC 
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Products  similar  to  Figure  18  would  be  very  useful  to  practitioners  as  well  as  for  further 
refinement  of  the  ERS  architecture,  toolset,  and  databases 


Set  of  Example  Cases  Available 

In  order  to  remove  the  "barrier"  to  entry  and  application  of  tradespace  approaches,  it  is 
important  to  have  example  cases  available  to  the  community.  This  provides  not  only  concrete 
representations  of  potential  outputs,  as  well  as  intermediate  artifacts,  from  a  tradespace 
exploration  activity,  but  also  helps  to  ground  abstract  processes  that  can  be  found  described  in 
many  policy  and  research  documents.  Additionally,  eventually  a  "template"  approach  can  be 
used  to  save  time  for  future  applications,  providing  for  a  standardized  representation  for  a  TSE 
study,  as  well  as  its  interpretation.  The  example  cases  should  include  not  only  end-to-end  TSE 
studies,  but  also  partial  TSE  studies  to  demonstrate  what  could  be  accomplished  with  limited 
time  and  resources  for  such  a  study.  It  is  essential,  however,  to  also  show  application  to  different 
scales  of  system  so  as  to  appreciate  the  relationship  of  level  of  detail  to  scale  of  system  and 
decision.  This  implies  that  having  cases  categorized  by  ACAT  level,  service,  domain  (e.g.,  land, 
sea,  air,  space),  and  so  forth,  would  help  build  a  body  of  knowledge  and  promote  coherence  in 
study  representations  and  outputs.  The  ERS  architecture  plays  an  essential  role  in  supporting  the 
development  of  such  example  cases  and  knowledge  codification  for  sharing.  In  the  near  term, 
pilot  studies  could  be  used  to  populate  early  examples  of  partial  studies. 


Current  ERS  Architecture  in  Support  of  Knowledge  Capture 

As  proposed  by  the  ERS  architecture,  shown  in  Figure  20,  a  number  of  knowledge  artifacts  are 
currently  proposed.  These  include  lessons  (learned)  within  a  knowledge  repository,  project  (case 
examples)  within  a  project  database,  language  (ontology)  within  a  document  database,  models 
(performance)  within  a  model  data  store,  and  analysis  tools  (within  a  "Decision  Analyzer" 
toolset).  Each  of  these  represent  concepts  identified  above  as  important  pieces  to  facilitate  rapid 
studies  (to  meet  the  ERS  goals  of  faster  consideration  of  more  alternatives),  as  well  as  to  increase 
the  likelihood  of  reducing  wasted  effort  "reinventing  the  wheel".  The  proposed  artifacts  also  will 
serve  to  build  an  able-to-be-queried  knowledge  infrastructure  that  should  enable  expert 
knowledge  to  be  accessible  to  less  experienced  users. 
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Figure  20  Annotated  ERS  Architecture  Overview  (from  Holland  2014) 


Moving  Forward 

A  number  of  next  steps  were  identified  over  the  course  of  this  research  which  would  enhance 

and  enable  the  ERS  vision,  specifically  as  related  to  tradespace  exploration  activities. 

•  Efforts  to  begin  compiling  appropriate  knowledge  relative  to  the  core  constructs  identified 
above  including  past  needs,  contexts,  constraints,  design  space,  performance  space,  value 
space,  performance  model,  value  model,  lessons  learned,  language,  case  examples,  and  tools. 

•  Research  into  effort  vs.  confidence  tradeoffs  so  that  projects  can  scale  effort  on  various 
activities  within  the  TSE  process  to  match  their  needs  subject  to  available  resources 

•  Development  of  fidelity  tradeoff  guidance  and  associated  tools  so  that  studies  can  scale  TSE 
implementation  appropriately 

•  Explicit  incorporation  of  resilience-related  ilities  evaluation  into  the  ERS  architecture, 
including  model  libraries  and  decision  analyzer  toolset 

•  Inclusion  of  value  models  along  with  performance  models  in  the  model  data  store 

•  Continued  piloting  of  parts  of  the  ERS  associated  TSE  processes,  as  well  as  full  end-to-end 
studies 

•  Continued  community  building  by  the  ERS  program,  including  offices  such  as  the  various  A9 
(e.g.  AF/A9,  OAS,  AFSPC/A9,  etc.)  and  other  entities  with  responsibilities  overlapping  with 
proposed  ERS  vision  and  capabilities 

•  Further  research  into  supporting  enabling  methods,  processes,  and  tools  that  can  facilitate 
TSE  knowledge  capture  and  reuse,  as  well  as  resource-effective  studies  that  can  quantify  and 
identify  resilient,  high  value  system  solutions  in  diverse  applications 
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Conclusions 


This  research  is  a  first  attempt  at  gathering  expert  knowledge  and  synthesizing  emerging  ERS- 
related  research,  toward  a  goal  enabling  novices  to  have  expert-like  decision  capability  through 
encoded  knowledge  and  data-driven  tradespace  analysis  framework  and  integrated  tool  suite. 
The  resulting  knowledge  and  information  gained  in  this  study  contributes  to  various  ongoing 
efforts  across  the  systems  community.  The  results  will  be  used  to  directly  inform  work  on  two 
SERC  research  projects:  RT-122,  Interactive  Model  Centric  Systems  Engineering  (IMCSE)  and  RT- 
113  llities  Tradespace  and  Affordability  (ITAP).  Results  will  be  shared  with  the  SERC  community 
and  may  possibly  contribute  to  other  relevant  SERC  projects.  The  research  team  will  use  the 
results  of  this  investigation  in  its  continuing  work  in  tradespace  exploration  research.  The  results 
will  be  used  toward  the  development  of  a  publishable  paper  to  transfer  findings  to  the  broader 
systems  community.  Specific  findings  will  be  shared  with  the  ERS  sponsor  and  its  technical  team 
and  partners  in  appropriate  discussions,  workshops  and  research  exchanges  to  potentially 
influence  and  extend  the  ongoing  initiatives  in  support  of  the  ERS  vision.  While  this  research 
investigation  has  focused  on  tradespace  exploration,  the  approach  taken  in  this  research  project 
can  be  applied  to  other  areas  within  the  ERS  program. 
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